Hi there! I'm Rocky. It sounds like you have a solid foundation with StarRocks 4.0.4 on Kubernetes and an Iceberg-based data lake. Since you mentioned that CN already has

enable_spill=true

, you are likely looking to ensure that memory-intensive queries on your Iceberg tables run reliably without hitting OOM (Out of Memory) errors. In a Kubernetes environment with S3 and External Catalogs, here is a checklist and some advanced configurations to optimize your spilling and Iceberg query performance: 1. Verify Local Storage Configuration Even with

enable_spill=true

, StarRocks requires a local directory to buffer intermediate data before (optionally) sending it to S3. * Parameter: Ensure

spill_local_storage_dir

is configured in your

cn.conf

(via the StarRocks Operator's

starrocksCnSpec.config

section). * Kubernetes Tip: Make sure this directory is mounted to a high-performance disk (like an

emptyDir

on an SSD or a dedicated PVC). If the directory is missing or not writable, spilling will fail.

yaml
    # Example snippet for StarRocksCluster CRD
    starrocksCnSpec:
      config: |
        spill_local_storage_dir=/opt/starrocks/cn/spill
      storageSpec:
        # It's recommended to mount a dedicated volume for spilling
        • name: cn-spill
          mountPath: /opt/starrocks/cn/spill
          storageSize: 100Gi
          storageClassName: local-path # Or your cloud provider's SSD class

2. Enable Remote Spill to S3 (Recommended) Since you are already using S3, you can leverage Remote Spill (introduced in v3.3+). This allows StarRocks to spill data to S3 after the local disk reaches a certain threshold, which is perfect for very large joins or aggregations on Iceberg. * Create a Storage Volume:

sql
    CREATE STORAGE VOLUME s3_spill_vol
    TYPE = S3
    LOCATIONS = ("<s3://your-bucket/spill/>")
    PROPERTIES ("aws.s3.region" = "us-east-1");

* Enable Remote Spill Session Variables:

sql
    SET GLOBAL enable_spill_to_remote_storage = true;
    SET GLOBAL spill_storage_volume = 's3_spill_vol';

3. Iceberg-Specific Spill (v4.0 feature) StarRocks 4.0 introduced Spillable Writes for Iceberg. If you are performing

INSERT INTO

or

CREATE TABLE AS SELECT

(CTAS) into Iceberg tables: * Parameter:

enable_connector_sink_spill

(Default is

true

in BE/CN config). * This prevents OOMs during large-scale ingestion into Iceberg by spilling the buffered rows to disk before the final file commit. 4. Important Troubleshooting for v4.0.4 Since you are on v4.0.4, there are a few known issues fixed in later 4.0.x releases (up to 4.0.8+) and v4.1: * Bug Fixes: Recent updates fixed potential data loss in hash table aggregation spills and crashes in

SpillMemTableSink

. * Recommendation: If you encounter intermittent query failures or incorrect results during spilling, I strongly recommend upgrading to the latest stable 4.0.x (e.g., 4.0.8) or 4.1.x. * Metadata Stale Issues: There was a known issue (#65381) in early 4.0 versions where Iceberg queries could fail with "File does not exist" due to stale metadata. If you see this, you can manually run

REFRESH EXTERNAL TABLE <table_name>

or check your metadata cache settings. 5. Memory Thresholds Check your CN memory thresholds to ensure spilling triggers appropriately: * `query_pool_spill_mem_limit_threshold`: (Default 1.0) Spilling triggers when query pool memory is full. * `spill_mode`: Ensure it is set to

auto

(default) or

force

for testing. How can I help you further? Are you experiencing a specific error message, or would you like to see a sample configuration for your Kubernetes operator deployment? References • administration/management/resource_management/spill_to_disk.md