@Rocky facing high execution wall time (8.89s) and memory allocation (8.16 GB) on an external Delta Lake query returning 0 rows, despite a 100% NVMe data cache hit. Setup: • StarRocks 3.5.1 Shared-Data • 3 CN nodes (4 vCPU / 30GiB RAM each) Profile Metrics (Empty Set Query): • `QueryExecutionWallTime`: 8.89s • `QueryAllocatedMemoryUsage`: 8.162 GB • `ScanRanges`: 1183 • `RawRowsRead`: 0 • `ScanTime`: 10.4ms • `DataCacheReadDiskBytes`: 764.419 MB • `FSIOBytesRead`: 0.000 B • `DeltaLake.getScanFiles`: 616ms Hypothesis: Since reading 764MB from local NVMe takes fractions of a millisecond, the 8.8s delay and 8.1GB RAM spike are not disk I/O bottlenecks. Instead, it appears StarRocks is instantiating 1,183 separate Parquet Readers, generating 1,183 scan tasks, and pre-allocating memory/chunk buffers for all of them. The 12-core cluster spends ~8 seconds purely on memory allocation (

malloc

), reader initialization, and teardown just to check empty footers. Questions: 1. Is this hypothesis correct? Is the 8.1GB RAM spike and 8.8s execution time entirely the initialization/teardown overhead of spawning ~1,180 Parquet readers? 2. Are there any StarRocks session variables (e.g., controlling chunk sizes, scan operators, or reader memory pre-allocation) that can mitigate this overhead on external catalogs?

Hello, I'm Rocky. Your hypothesis is largely correct. The 8.8s wall time and 8.1GB memory spike for a query returning 0 rows are primarily driven by the initialization and buffer pre-allocation overhead of spawning 1,183 Parquet readers across a relatively small 12-core cluster (3 CNs × 4 vCPUs). 1. Hypothesis Confirmation The metrics suggest that even though the Data Cache hit was 100%, the execution engine still had to: * Parse 1,183 Parquet Footers: The

DataCacheReadDiskBytes

(764 MB) confirms that StarRocks read approximately 645 KB per file/range from the local NVMe cache. This is the size of the Parquet metadata/footers. * Allocate Buffers for each Reader: StarRocks pre-allocates memory for "Chunks" based on the

chunk_size

variable. At 8.16 GB for 1,183 readers, each reader is consuming roughly 6.9 MB of memory. This includes the reader object, metadata structures, and pre-allocated column buffers for the scan. * Task Management Overhead: Processing 1,183 separate scan tasks on a cluster with only 12 vCPUs total means each core is handling roughly 100 tasks. The 8.8s is spent on the overhead of

malloc

, reader setup, metadata processing, and context switching for these 1,183 tasks. 2. Recommended Session Variables You can mitigate this overhead by reducing the memory footprint of each reader and limiting the concurrency of the initialization phase. Try setting the following variables for the session: *

set chunk_size = 1024;

(Default: 4096) This is the most effective way to reduce memory allocation. It reduces the size of the pre-allocated buffers for each column in every Parquet reader. If the table has many columns, setting this to 1024 or even 512 can significantly drop the memory spike. *

set pipeline_max_read_concurrency = 4;

(Default: 0/Dynamic) This limits the number of concurrent read tasks the pipeline engine will schedule. Reducing this prevents the cluster from trying to instantiate and manage too many readers simultaneously. *

set connector_io_tasks_per_scan_operator = 4;

(Default: 16) This reduces the number of concurrent I/O threads per scan operator. Since you are hitting the NVMe cache, you don't need high I/O concurrency, and lowering this saves memory. *

set enable_connector_adaptive_io_tasks = false;

Disabling the adaptive logic allows you to strictly enforce the lower

connector_io_tasks_per_scan_operator

limit. *

set enable_parquet_reader_page_index = false;

set enable_parquet_reader_bloom_filter = false;

If your Parquet files contain large Page Indexes or Bloom Filters, the reader will load these into memory during initialization. If your query filters are simple or no data is being returned, disabling these can reduce the per-reader memory footprint. *

set parallel_fragment_exec_instance_num = 1;

This forces fewer scan instances per node, which can help reduce the management overhead when dealing with a high number of small scan ranges. Summary Checklist If the table has a very high column count (e.g., 500+ columns), even a "0 row" query will be expensive because the reader must allocate metadata for every column. To further optimize, ensure you are only selecting the specific columns needed rather than using

SELECT *

. References • sql-reference/System_variable.md • using_starrocks/Cost_based_optimizer.md • knowledge/usage/best-practice.md • data_source/catalog/iceberg/iceberg_catalog.md