@Rocky WHat's idea size for ingesting parquet to StarRocks in iceberg table? I'm using is4gen.4xlarge EC2 as CN

For ingesting data into an Iceberg table (as a sink) using StarRocks, the ideal Parquet file size is generally between 256 MB and 512 MB. Recommended Sizes & Configuration * Ideal Size: Aim for 256 MB to 512 MB. * Under 128 MB: Leads to "small file" problems, increasing metadata overhead and slowing down query planning. * Over 1 GB: Can reduce the effectiveness of data pruning (min/max skipping) because each file covers too wide a range of values. * Session Variable (v3.3+): You can control the target file size for Iceberg sinks using the session variable:

sql
    SET connector_sink_target_max_file_size = 268435456; -- 256 MB in bytes

* Compaction: If you already have small files, use StarRocks' Iceberg maintenance procedure to merge them:

sql
    ALTER TABLE iceberg_table EXECUTE rewrite_data_files("min_file_size_bytes"="134217728"); -- Merges files < 128MB

Considerations for

is4gen.4xlarge

CN The

is4gen.4xlarge

instance is a storage-optimized instance (16 vCPUs, 128 GB RAM) with high-performance local NVMe SSDs. While Iceberg ingestion (sink) is primarily limited by S3/network throughput, you should leverage your hardware for query performance: * Data Cache: Since

is4gen

has fast NVMe storage, enable the StarRocks Data Cache. This will cache Iceberg Parquet data from S3 onto your local NVMe disks, providing near-native performance for subsequent queries. * Set

block_cache_enable = true

in your

cn.conf

. * Point

block_cache_disk_path

to your NVMe mount point. * Parallelism: With 16 vCPUs, you can handle significant write parallelism. Ensure your

INSERT INTO

or

INSERT OVERWRITE

jobs aren't throttled by low session parallelism. Ingestion Best Practices 1. Batch Ingestion: Avoid high-frequency micro-batches (e.g., every few seconds) when sinking to Iceberg, as this creates too many small files and manifest updates. 2. Partitioning: If your Iceberg table is partitioned, StarRocks writes at least one file per partition per writer. If you have too many partitions, this can naturally lead to small files. Try to ingest data partition-by-partition if you are loading large volumes of historical data. 3. Memory:

is4gen.4xlarge

has 128GB of RAM. If you encounter memory issues during large sinks, you can enable "spill to disk" for the load job:

sql
    SET enable_spill = true;

References • unloading/Spark_connector.md • loading/loading_introduction/loading_considerations.md • loading/Spark-connector-starrocks.md