Hi, can anyone please explain why total time here is ~11secs yet the items under total only add up to ~3secs? This is an INSERT INTO table SELECT FROM FILES(), presumably time is taken for insert but not accounted for in profile?

MySQL  localhost:9030  -  SQL > ANALYZE PROFILE FROM '019adaa2-2f45-72f0-8147-d687cb5fcc74';
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String                                                                                                                                         |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Summary                                                                                                                                        |
|     Attention: The transaction of the statement will be aborted, and no data will be actually inserted!!!                      |
|     QueryId: 019adaa2-2f45-72f0-8147-d687cb5fcc74                                                                                              |
|     Version: 4.0.1-cd9df36                                                                                                                     |
|     State: Finished                                                                                                                            |
|     TotalTime: 10s899ms                                                                                                                        |
|         ExecutionTime: 2s840ms [Scan: 2s563ms (90.25%), Network: 0ns (0.00%), ResultDeliverTime: 270.747ms (9.53%), ScheduleTime: 0ns (0.00%)] |
|         CollectProfileTime: 0                                                                                                                  |
|         FrontendProfileMergeTime: 1.212ms                                                                                                      |
|     QueryPeakMemoryUsage: ?, QueryAllocatedMemoryUsage: 110.682 MB                                                                             |
|     Top Most Time-consuming Nodes:                                                                                                             |
|         1. FILE_SCAN (id=0) : 2s563ms (99.70%)                                                                                            |
|         2. OLAP_TABLE_SINK: 8.272ms (0.32%)                                                                                                    |
|     Top Most Memory-consuming Nodes:                                                                                                           |
|     NonDefaultVariables:                                                                                                                       |
|         big_query_profile_threshold: 30s -> 10s                                                                                                |
|         enable_adaptive_sink_dop: false -> true                                                                                                |
|         enable_load_profile: false -> true                                                                                                     |
|         enable_local_shuffle_agg: true -> false                                                                                                |
|         enable_materialized_view_rewrite: true -> false                                                                                        |
|         enable_profile: false -> true                                                                                                          |
|         use_compute_nodes: -1 -> 0                                                                                                             |
| Fragment 0                                                                                                                                     |
| │   BackendNum: 1                                                                                                                                |
| │   InstancePeakMemoryUsage: 55.454 MB, InstanceAllocatedMemoryUsage: 110.682 MB                                                                 |
| │   PrepareTime: ?                                                                                                                               |
| └──OLAP_TABLE_SINK                                                                                                                               |
|    │   TotalTime: 8.272ms (0.32%) [CPUTime: 8.272ms]                                                                                             |
|    │   OutputRows: 7.751K (7751)                                                                                                                 |
|    │   PartitionType: RANDOM                                                                                                                     |
|    │   Table: temp_66a08bf2-d977-4e62-9141-afb39033636f                                                                                          |
|    └──FILE_SCAN (id=0)                                                                                                                       |
|           Estimates: [row: 1, cpu: ?, memory: ?, network: ?, cost: 0.0]                                                                   |
|           TotalTime: 2s563ms (99.70%) [CPUTime: 391.850us, ScanTime: 2s563ms]                                                             |
|           OutputRows: 7.751K (7751)                                                                                                       |
|           Detail Timers: [ScanTime = IOTaskExecTime + IOTaskWaitTime]                                                                     |
|               IOTaskExecTime: 2s563ms                                                                                                     |
|                   FileScanner:                                                                                                            |
|                       FileReadTime: 2s530ms                                                                                               |
|                   ScannerTotalTime: 2s562ms                                                                                               |
|               IOTaskWaitTime: 75.310us                                                                                                    |
|                                                                                                                                                    |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+

The ~11 sec should be the time it takes for the front end to receive the query, schedule, commit, and publish version. The execution time (~3s) is the actual time consumed for this query (scanning and io) in the backend.

Ok, thanks, but 8s seems a long time to process a relatively small insert. What can be done to speed this up?

Seems like the profile is incomplete, can you share the complete one

curl -u${username}:${password} "${fe_ip}:${fe_http_port}/query_profile?query_id=019adaa2-2f45-72f0-8147-d687cb5fcc74"

Thanks, I ended up asking AI, passing it the logs for the period and the query profile. Apparently the main contributors are FE-side orchestration + transaction commit + publish + RPC coordination, along with Automatic table statistics collection