@Rocky Im using starrocks version 4.1.1 and I got some parquet files in s3 and I use pipe Files() in order to load it to starRocks table. I was wondering if starrocks has the capability like trino to say here is the parquet files commit it as a iceberg data into the iceberg s3 table

Yes, StarRocks supports this capability via the

add_files

procedure. This allows you to "commit" existing Parquet or ORC files from an S3 location into an Iceberg table by updating the table's metadata, without actually rewriting or moving the data files. This is equivalent to the

system.add_files

procedure found in Trino or Spark. How to use it To register your Parquet files into an existing Iceberg table, use the following syntax within your Iceberg catalog:

sql
ALTER TABLE [catalog.][database.]table_name
EXECUTE add_files(
    location = '<s3://your-bucket/path/to/parquet_files/>',
    file_format = 'parquet',
    recursive = true
);

Key Parameters: *

location

: The S3 path where your Parquet files are stored. *

file_format

: The format of the files (currently

parquet

or

orc

). *

recursive

: (Optional) Whether to scan subdirectories. Defaults to

true

. Alternatives If you are trying to register an entire existing Iceberg table (including its existing metadata files), you can use the

register_table

procedure instead:

sql
CALL system.register_table(
    database_name = 'your_db',
    table_name = 'your_table',
    metadata_file = '<s3://your-bucket/path/metadata/v1.metadata.json>'
);

Important Notes: * Schema Matching: Ensure that the schema of the Parquet files in S3 matches the schema of the Iceberg table. StarRocks does not perform complex schema evolution/validation during this metadata-only operation. * Privileges: You must have the appropriate privileges (at least

INSERT

or

ALTER

) on the target Iceberg table and catalog. * Version: This feature is available in the version you are using (it was introduced in the 3.x/4.x release line). For more details, you can refer to the Iceberg Procedures section of the StarRocks documentation. References • data_source/catalog/iceberg/procedures.md • data_source/catalog/iceberg/iceberg_meta_table.md • data_source/catalog/iceberg/iceberg_catalog.md

@Rocky can it be done via pipe Files() ?

No, you cannot use

Pipe

or the

FILES()

table function to perform the metadata-only "commit" operation (similar to Trino's

add_files

). Here is the distinction in StarRocks: 1.

add_files

(Metadata Only - What you want) If you want to register existing Parquet files into an Iceberg table without rewriting the data, you must use the

add_files

procedure. This is the direct equivalent to Trino’s

system.add_files

. * Mechanism: It updates the Iceberg manifest files to include the existing S3 paths. * Syntax:

sql
    ALTER TABLE iceberg_catalog.db.table
    EXECUTE add_files(location = '<s3://path/to/files/>', file_format = 'parquet');

2.

Pipe

and

FILES()

(Data Ingestion - Not Metadata Only) The

Pipe

object in StarRocks is designed for continuous data loading. It uses an

INSERT INTO ... SELECT FROM FILES()

statement under the hood. * Mechanism: StarRocks reads the records inside your Parquet files and writes new data files into the Iceberg table's storage location. * Result: This results in duplicated data in S3 (the original files plus the new Iceberg-managed files) and consumes compute resources for processing the records. * Syntax (This will not do a metadata-only commit):

sql
    CREATE PIPE my_pipe AS
    INSERT INTO iceberg_table
    SELECT * FROM FILES("path" = "s3://...", "format" = "parquet");

Summary If your goal is to "point" Iceberg to existing files to avoid data movement/rewriting, you must use

ALTER TABLE ... EXECUTE add_files

.

Pipe

is the right tool if you want to continuously monitor an S3 bucket and ingest new data into the Iceberg table as it arrives, but it will always involve a write operation. References • data_source/catalog/iceberg/procedures.md • sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md • loading/s3.md • sql-reference/sql-functions/table-functions/files.md References • sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md • loading/hdfs_load.md • sql-reference/information_schema/pipe_files.md • data_source/catalog/iceberg/procedures.md