@Rocky what is recommended way to load data into starrocks managed iceberg tables ?

The recommended way to load data into Iceberg tables managed by StarRocks (tables created within an Iceberg Catalog) depends on your data source and volume. StarRocks treats these tables similarly to native tables for DML operations. 1. Batch Loading from Files (Recommended for S3/HDFS) For data residing in cloud storage or HDFS, the most efficient way is using the

FILES()

table function with an

INSERT INTO

statement.

sql
INSERT INTO iceberg_catalog.db_name.table_name
SELECT * FROM TABLE(
    FILES(
        "path" = "<s3://bucket/data/*.parquet>",
        "format" = "parquet"
    )
);

* Why: It leverages StarRocks' distributed execution engine to pull and sink data directly without intermediate storage. * Note: Use

INSERT OVERWRITE

if you need to replace existing data. 2. ETL from Other Tables If the data is already in StarRocks (internal tables) or another external catalog (e.g., Hive, Hudi), use a standard

INSERT INTO ... SELECT

.

sql
INSERT INTO iceberg_catalog.db_name.dest_table
SELECT * FROM internal_db.source_table;

3. Metadata-Only Ingestion (

add_files

) If your data files are already in the correct Iceberg-compatible storage location but not yet part of the table metadata, you can use the

add_files

procedure. This is a "zero-copy" load.

sql
ALTER TABLE iceberg_catalog.db_name.table_name
EXECUTE add_files(location = '<s3://bucket/path/to/files/>', file_format = 'parquet');

4. Real-time / High-Volume Stream (Flink/Spark) For continuous ingestion or massive batch jobs from external compute: * Flink Connector: Recommended for real-time streaming pipelines. It supports sinking data directly into Iceberg catalogs via StarRocks. * Spark Connector: Recommended for heavy ETL workloads in Spark that need to write to Iceberg tables managed by StarRocks. Best Practices * Avoid `INSERT INTO ... VALUES`: Do not use this for production loads as it is not optimized for large volumes and can create many small files. * Partition-by-Partition: When ingesting large historical datasets into partitioned tables, it is recommended to load data partition by partition. This helps avoid generating too many small files and reduces memory pressure on the writers. * Table Properties: Ensure your Iceberg table is created with

file_format = "parquet"

(currently the only supported write format) and consider setting

compression_codec = "zstd"

for better storage efficiency. * Version Support: DML for Iceberg (INSERT/OVERWRITE) requires StarRocks v3.1 or higher. Sorting support (

ORDER BY

in DDL) was added in v4.0. References • data_source/catalog/iceberg/procedures.md • data_source/catalog/iceberg/DDL.md • quick_start/iceberg.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/iceberg/iceberg_catalog.md