@Rocky, is there a process to follow to migrate from native storage to iceberg for tables?

Hi there! I'm Rocky, and I can definitely help you with that. Migrating data from StarRocks native (internal) storage to Apache Iceberg is a straightforward process, especially with the enhancements introduced in version 3.1 and later. Since StarRocks can act as both a source and a sink for Iceberg, you can perform the migration using standard SQL. Here is the process to follow: 1. Create an Iceberg Catalog First, you need to set up an external catalog that points to your Iceberg storage (e.g., S3, HDFS) and metastore (e.g., Hive Metastore, AWS Glue, or REST).

sql
CREATE EXTERNAL CATALOG iceberg_catalog
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "HIVE", -- or GLUE, REST, etc.
    "iceberg.catalog.hive.metastore.uris" = "thrift://<hms_ip>:9083"
);

2. Create the Destination Iceberg Table You have two primary ways to create the table in the Iceberg catalog: * Option A: CREATE TABLE AS SELECT (CTAS) This is the simplest way to migrate both schema and data in one command.

sql
    CREATE TABLE iceberg_catalog.target_db.iceberg_table
    AS SELECT * FROM default_catalog.source_db.native_table;

* Option B: Manual Create followed by INSERT Use this if you need specific Iceberg table properties or partitioning.

sql
    -- Create the table first
    CREATE TABLE iceberg_catalog.target_db.iceberg_table (
        id BIGINT,
        name STRING,
        dt DATE
    )
    PARTITION BY (dt)
    PROPERTIES (
        "compression" = "zstd"
    );

    -- Load the data
    INSERT INTO iceberg_catalog.target_db.iceberg_table
    SELECT * FROM default_catalog.source_db.native_table;

3. Key Considerations for Migration * Version Requirement: Writing to Iceberg catalogs via

INSERT INTO

or

CTAS

requires StarRocks v3.1 or later. * Data Types: Ensure the data types in your native table are compatible with Iceberg. Most standard types (INT, BIGINT, STRING, DECIMAL, etc.) map directly. * Partitioning: If your native table is partitioned, you should typically define corresponding partitioning in the Iceberg table to maintain performance. Iceberg supports "Hidden Partitioning," which StarRocks can leverage. * Default Values: When creating an Iceberg table manually in StarRocks, non-partition columns must often be defined with

DEFAULT "NULL"

as per Iceberg specification requirements in the StarRocks connector. * Parquet Format: Currently, StarRocks supports sinking data to Iceberg tables primarily in Parquet format. 4. Verification After the migration, you can verify the data by querying the new Iceberg table:

sql
SELECT count(*) FROM iceberg_catalog.target_db.iceberg_table;

If you are looking to migrate a very large volume of data, you can also use

INSERT OVERWRITE

for specific partitions to make the process more manageable or to perform the migration in batches. Let me know if you need help with a specific catalog configuration! References • data_source/External_table.md • data_source/catalog/iceberg/iceberg_catalog.md