@Rocky we are using starrocks 4.1 shared database with an iceberg external catalog on s3 table buckets. does starrocks support merge on read updates?

Yes, StarRocks 4.1 supports Merge-on-Read (MoR) for Iceberg V2 tables within an external catalog. This support covers both reading existing MoR tables and writing updates/deletes using the MoR model. Key MoR Capabilities in StarRocks 4.1: * Reading MoR Tables: StarRocks can query Iceberg V2 tables that contain positional deletes and equality deletes. The engine's vectorized reader efficiently merges these delete files during the scan process to return the correct results. * Writing Updates/Deletes: When you perform

UPDATE

or

DELETE

operations on an Iceberg V2 table via an external catalog, StarRocks follows the Merge-on-Read model. Instead of rewriting the entire data file (Copy-on-Write), it generates positional delete files and commits them to the Iceberg metadata. * S3 & Shared-Data Architecture: This functionality is fully compatible with your setup using AWS S3 and StarRocks' shared-data (cloud-native) architecture. In version 4.1, StarRocks has further optimized the metadata handling and caching for shared-data clusters to reduce latency when accessing external catalogs on object storage. Version Context: * Iceberg V2 Parquet: Supported for both reads and writes since v3.1. * Iceberg V2 ORC: Supported since v3.0. * StarRocks 4.1 Improvements: Specifically focuses on adaptive layout and multi-tenant operability for shared-data clusters, ensuring that as your Iceberg-backed "shared databases" grow, the system can better manage tablet distribution and resource isolation. Implementation Notes: * Metadata Cache: Ensure

enable_iceberg_metadata_cache

is enabled (it is

true

by default in recent versions) to avoid frequent S3 lookups for manifest and delete files. * Catalog Metastore: Whether you are using AWS Glue, Hive Metastore, or a REST catalog (like Tabular or Polaris), the MoR behavior remains consistent as long as the underlying table is configured as Iceberg V2. For the best performance in your 4.1 shared-data environment, it is recommended to use AWS Glue or an Iceberg REST catalog, as these integrate most seamlessly with StarRocks' credential-vending and metadata polling mechanisms. References • administration/management/monitoring/metrics-shared-data.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md • quick_start/shared-data.md

@Rocky my compute node is experiencing exit code 137when executing an insert into statement from one iceberg table into another

The exit code 137 indicates that your Compute Node (CN) process was force-killed by the Operating System or a container orchestrator (like Kubernetes) with a

SIGKILL

. This is almost always the result of an Out of Memory (OOM) event where the process exceeded the memory limit assigned to the pod or host. When executing an

INSERT INTO

between Iceberg tables, StarRocks must buffer data in memory before writing it to S3 as Parquet files. Here are the most likely causes and how to fix them: 1. High Number of Concurrent Partition Writers (Most Common) If your source data is not ordered by the destination partition key, every Compute Node may attempt to write to every partition simultaneously. Each open partition writer requires its own memory buffer (typically 100MB+ per writer). * Fix: Add an

ORDER BY

clause to your

SELECT

statement using the partition columns of the destination table. This ensures each Compute Node handles only a few partitions at a time, drastically reducing memory overhead.

sql
    INSERT INTO target_iceberg_table
    SELECT * FROM source_iceberg_table
    ORDER BY partition_column; -- Helps cluster data to fewer writers

2. Spilling Configuration StarRocks 4.x supports spilling for Iceberg sinks to prevent OOMs when memory is tight. Ensure this is enabled (it should be

true

by default): * Check the BE/CN configuration or session variable:

enable_connector_sink_spill = true

. * If your disk is slow or full, spilling might not happen fast enough to prevent the OS from killing the process. 3. High Query Parallelism If

parallel_fragment_exec_instance_num

(or

pipeline_dop

in older versions) is set too high, StarRocks will spawn many concurrent write "instances." Each instance maintains its own buffers. * Fix: Try reducing the parallelism for this specific session:

sql
    SET parallel_fragment_exec_instance_num = 4; -- Or a lower value
    INSERT INTO target_iceberg_table SELECT ...;

4. Memory Limits Mismatch If you are running in Kubernetes, your Pod's memory limit might be set lower than StarRocks' internal

mem_limit

. * Check `be.conf`: The

mem_limit

parameter defaults to 90% of the host's memory. If your K8s limit is 16GB but StarRocks thinks it has 32GB, the OS will kill the process before StarRocks' own memory management can kick in. * Action: Ensure the K8s memory limit is strictly higher than the

mem_limit

configured in StarRocks. Troubleshooting Steps: 1. Check OS/K8s Logs: Run

dmesg -T | grep -i oom

on the host or

kubectl describe pod <cn-pod>

to confirm it was an "OOMKilled" event. 2. Review Query Profile: If possible, check the query profile in the StarRocks FE (via

show query profile "/"

) to see which operator (e.g.,

IcebergTableSink

) was consuming the most memory. 3. Monitor Memory Trackers: You can view real-time memory usage by hitting the CN's HTTP endpoint:

http://<cn_host>:<cn_http_port>/mem_tracker

. Summary Recommendation: The most effective fix is usually sorting the source data by the partition key in your

INSERT

statement to reduce the number of active writers. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • loading/Flink-connector-starrocks.md

@Rocky where do i check and set enable_connector_sink_spill

In StarRocks 4.x,

enable_connector_sink_spill

is a BE/CN configuration parameter (not a session variable) that controls whether writes to external tables (like Iceberg) can spill to disk when memory is low. 1. How to Check the Current Value Because this is a Backend (BE) or Compute Node (CN) configuration, you check it via the node's HTTP interface rather than through SQL. * Via Web Browser or Curl: Access the

/var

or

/conf

endpoint of your BE or CN nodes:

bash
    curl http://<BE_IP>:<BE_HTTP_PORT>/var | grep enable_connector_sink_spill

_(Default

BE_HTTP_PORT

is

8040

unless changed in your config)._ 2. How to Set/Modify It Since the parameter is mutable, you can change it dynamically without restarting the nodes, though you should also update the configuration file for the change to persist. Dynamic Change (No Restart) You can use the

update_config

API on each BE/CN node:

bash
curl -X POST "http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?enable_connector_sink_spill=true"

Persistent Change (Requires Restart for the file to be read) Add or modify the line in your

be.conf

or

cn.conf

file:

properties
enable_connector_sink_spill = true

Important Requirements For this feature to work correctly in your Iceberg setup: 1. General Spilling: Ensure the session variable

enable_spill

is also set to

true

for your query:

sql
    SET enable_spill = true;

2. Spill Directory: You must have a local spill directory configured in your `be.conf`/`cn.conf`. Check for the

spill_local_storage_dir

parameter. If it's not set, spilling cannot occur.

properties
    spill_local_storage_dir = /data/starrocks/spill

3. Iceberg Sink: As of version 4.0/4.1, this specific sink spilling feature is primarily supported for Iceberg tables to prevent OOMs during large

INSERT INTO

operations. References • administration/management/BE_configuration.md • administration/management/resource_management/spill_to_disk.md • sql-reference/System_variable.md References • sql-reference/System_variable.md • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md

@Rocky this is running on k8 on eks with these setting: storage_root_path = /mnt/nvme/storage;/mnt/ebs/storage spill_local_storage_dir = /mnt/ebs/spill datacache_mem_size = 50% datacache_disk_size = 85% starlet_star_cache_disk_size_percent = 85 enable_datacache_disk_auto_adjust = false - Time: 2026-08-05 215201–215227 UTC - Query ID: 019fd3e9-7bae-7906-b30e-bcd208d85077 - State: ERR - Duration: 26.018s - Scan: 0 rows / 0 bytes - FE audit ErrorCode: empty — it records that the statement failed but not the error text. connector:SpillPartitionChunkWriter:_merge_chunks() → _spill() → ConnectorChunkSink::finish() INSERT starts → connector sink spills / merges large caption chunks → 1.36 GiB contiguous BinaryColumn allocation on CN-3 → statement returns ERR after ~26s → CN-3 is SIGKILLed about two minutes later

Based on your trace and configuration, the issue is a memory exhaustion event during the "Spill Merge" phase of your Iceberg write. The specific error occurred in

SpillPartitionChunkWriter::_merge_chunks()

. This component is responsible for taking the temporary "spilled" data and merging it into a final format to write to S3/Iceberg. Why it's failing 1. Massive Contiguous Allocation: You noted a 1.36 GiB contiguous BinaryColumn allocation. In StarRocks,

BinaryColumn

stores variable-length data (strings, JSON, or blobs). A single 1.3GB allocation is extremely large for a vectorized execution engine and suggests your source data has very wide rows or very long string values. 2. Memory Resource Conflict: You have

datacache_mem_size = 50%

. On a 64GB node, 32GB is strictly reserved for the Data Cache. This leaves only 32GB for the OS, the StarRocks process overhead, and all active query buffers. When the query engine tries to grab a contiguous 1.36GB block while other buffers are active, it can easily hit the

mem_limit

or trigger a Kubernetes OOM. 3. Spill Merge Limit: There is a BE configuration called

load_spill_memory_usage_per_merge

which defaults to 1 GB. Your trace shows an allocation of 1.36 GiB, which exceeds this default and likely caused the internal failure (

ERR

) before the OS eventually killed the process (

SIGKILL

). Recommended Fixes 1. Adjust Data Cache (Immediate Action) For nodes performing heavy ETL/Iceberg writes, 50% memory for cache is too high. It starves the query engine of the "workspace" memory needed for large writes. * Action: Reduce

datacache_mem_size

to 20% or 30% in your

cn.conf

.

properties
    datacache_mem_size = 20%

2. Increase Spill Merge Memory limit If you have large rows, you need to allow the merger to use more memory, or the query will fail even if the node has RAM. * Action: Increase the threshold in

cn.conf

(or via

curl

update_config):

properties
    load_spill_memory_usage_per_merge = 2147483648  # Increase to 2GB

3. Optimize the SQL Statement To prevent the

SpillPartitionChunkWriter

from having to merge such massive chunks, reduce the number of active writers and sort the data so "like" rows are processed together. * Action: Add an

ORDER BY

clause to your

INSERT

statement using the destination table's partition columns.

sql
    INSERT INTO target_iceberg_table
    SELECT * FROM source_iceberg_table
    ORDER BY partition_col1, partition_col2;

4. K8s Pod Limit Alignment Ensure your Kubernetes memory limit is at least 10–15% higher than the StarRocks

mem_limit

. * If your Pod limit is 64Gi, set

mem_limit

in

cn.conf

to

56G

(or ~85-90%). If the K8s limit and StarRocks limit are too close, the K8s OOM Killer will

SIGKILL

(137) the pod before StarRocks can trigger its own "Soft Limit" protective measures (like canceling the query gracefully). Summary of settings to check/change: Parameter | Suggested Value | Note

datacache_mem_size

|

20%

| Gives more room for

INSERT

buffers.

load_spill_memory_usage_per_merge

|

2G

| Prevents error on large

BinaryColumn

merges.

connector_sink_compression_codec

|

LZ4

| (v4.1) Ensures chunks are compressed in memory.

enable_spill

|

true

| (Session variable) Must be set for the query. Why the 2-minute delay before the SIGKILL? The query failed immediately because the internal allocation exceeded a threshold (returning

ERR

). However, the process likely held onto fragmented memory or remained in a "High Memory Pressure" state. Kubernetes or the Linux OOM killer eventually decided the process was unhealthy and killed it. Reducing the

datacache_mem_size

is the most effective way to provide the "breathing room" needed to avoid this. References • administration/management/BE_configuration.md