@Rocky - Does Starrocks distributes the parquet files of Iceberg table by size during read side? We're observing that the the distribution is not having by size.

Yes, StarRocks distributes the workload of reading Iceberg Parquet files by size, but the behavior depends on how the files are divided into "splits" (ScanRanges) and how those splits are scheduled. If you are observing an imbalanced distribution, it is likely due to how the FE (Frontend) handles split planning or how the pipeline engine's morsels are assigned. 1. How StarRocks Handles Iceberg Reading StarRocks uses a split-based scanning mechanism for external tables like Iceberg: * Split Logic: For large Parquet files, StarRocks' FE divides them into multiple ScanRanges (typically based on a target size like 128MB or 256MB). This allows multiple BE/CN nodes to process different parts of the same large file in parallel. * Small Files: If your Parquet files are smaller than the split threshold (e.g., many 10MB files), StarRocks cannot split them further. Each small file becomes a single ScanRange. * Distribution Policy: The FE scheduler attempts to distribute these ScanRanges across available BE nodes. Traditionally, the scheduler balances the number of ScanRanges per node rather than the aggregate bytes per node. If your table has a mix of large (split) files and many tiny files, a node assigned 10 small files will finish much faster than a node assigned 10 large splits, leading to the "not distributed by size" behavior you observed. 2. Adaptive Execution (Morsels) StarRocks uses a morsel-driven pipeline engine. While the FE creates the initial ScanRanges, the BEs process data in "morsels" (small work packages). * In theory, if one BE finishes its assigned work early, the pipeline engine is designed to handle more work. However, for external tables, the ScanRanges are often "pinned" to specific nodes during the planning phase to maintain connection/cache efficiency. * If you see a "long tail" in your query profile (where a few nodes take much longer than others), it confirms that the initial split assignment was imbalanced. 3. Tuning and Verification To investigate or improve the distribution: * Check Query Profile: Run

EXPLAIN ANALYZE <sql>

and look at the

OLAP_SCAN

or

ICEBERG_SCAN

node. Compare

BytesRead

and

ScanTime

across different BE nodes. If

BytesRead

varies significantly, the assignment is skewed. * Configure Split Size: * In recent versions (e.g., v4.0.4+), you can use the session variable

connector_huge_file_size

to adjust the split threshold for Iceberg. * Check if your Iceberg table properties (like

read.split.target-size

) are set, as StarRocks often respects these metadata hints. * Small File Compaction: If the imbalance is caused by a "storm of small files," the best fix is to run Iceberg compaction (using StarRocks'

system.rewrite_data_files

procedure or Spark/Flink maintenance jobs) to merge tiny files into uniform 128MB–512MB files. Summary: StarRocks distributes by "splits" of a target size. If your data consists of many non-uniform small files, the distribution will appear to be by file count rather than actual size. Tightening the split size or compacting small files is the standard solution. References • sql-reference/System_variable.md