@Rocky Big CTAS into Iceberg (600M rows, month partition + ORDER BY) OOMKills the CN instead of failing the query — seems Parquet writer allocations leak outside MemTracker. Any way to guarantee graceful query failure?

The issue you're describing—where a massive CTAS into Iceberg with

ORDER BY

and partitioning causes a Compute Node (CN) OOMKill instead of a query failure—is typically caused by the "buffered" nature of Parquet writers and the interaction between the sort operator and the sink. When you have 600M rows and a high number of partitions (e.g., month-based), each partition writer maintains its own set of memory buffers (for row groups and column pages). If the query plan involves high parallelism (

pipeline_sink_dop

) and many partitions are being written to concurrently, this memory footprint can balloon rapidly, often outpacing the `MemTracker`'s ability to trigger a graceful rollback. Here are the specific ways to mitigate this and ensure graceful query failure or successful completion: 1. Enable Connector Sink Spilling (StarRocks v4.0+) Starting in version 4.0.0, StarRocks introduced a dedicated mechanism to handle memory pressure during Iceberg writes. This allows the sink to spill data to disk when memory is low, preventing OOM. * BE Configuration: Set

enable_connector_sink_spill = true

(default in v4.0) in your `be.conf`/`cn.conf`. * This is the most direct solution for the "Parquet writer allocation" problem you've identified. 2. Throttle Sink Parallelism If you are on a version earlier than 4.0, or if spilling isn't sufficient, the most effective way to reduce memory is to limit the number of concurrent Parquet writers. Each writer holds significant column/page buffers. * Session Variable:

SET pipeline_sink_dop = 1;

(or a low number like 2 or 4). * By default, this is often set to the number of CPU cores, which can lead to hundreds of open Parquet writers across a cluster, each consuming several hundred MBs. 3. Ensure Strict Query Memory Limits To guarantee the query fails before the OS kills the process, you must leave a "safety buffer" between the

query_mem_limit

and the node's physical memory. * Session Variable:

SET query_mem_limit = <value_lower_than_CN_memory>;

* If the CN has 64GB, set

query_mem_limit

to 40GB or 48GB. This gives the system room to detect the limit and cancel the query before the Linux OOM Killer intervenes. 4. Optimize the

ORDER BY

and Partitioning Interaction The

ORDER BY

clause is helpful for Parquet file layout, but if the sort order does not align with the partition column, the sink may be forced to keep many partition writers open simultaneously. * Strategy: Ensure the first column in your

ORDER BY

is the partition column (the month column). This ensures that the data is sent to the sink "partition-by-partition," allowing StarRocks to close the writer for one month before opening the next, drastically reducing peak memory usage. 5. Adjust BE Memory Tracking Parameters If you suspect the load process is the culprit, you can tighten the global load memory limits: * BE Config:

load_process_max_memory_limit_percent = 30

(default). Lowering this to

20

or

15

will trigger backpressure and flushes earlier. * BE Config:

write_buffer_size

. While primarily for internal tables, reducing this can sometimes reduce the size of the intermediate chunks being handed off to the connector. Summary Checklist for Large Iceberg CTAS: 1. Upgrade to 4.0+ to use

enable_connector_sink_spill

. 2. Set

ORDER BY (partition_col, ...)

to keep the number of active writers low. 3. Lower

pipeline_sink_dop

to 1–4 to reduce concurrent buffer allocations. 4. Set a conservative

query_mem_limit

to ensure the query fails gracefully. References • sql-reference/System_variable.md • administration/management/BE_configuration.md • data_source/catalog/iceberg/iceberg_catalog.md