@Rocky in starrocks latest version, is there support already for the following commands for iceberg tables:

rewrite_manifests

,

expire_snapshots

and

remove_orphan_files

?

Yes, StarRocks supports these Iceberg maintenance commands in its recent versions (specifically starting from v3.2 for some and fully integrated with advanced arguments by v4.1). These operations are performed using the

ALTER TABLE ... EXECUTE

syntax. Here is the breakdown: 1.

expire_snapshots

This command is supported and is used to delete snapshots older than a specific timestamp and data files that are no longer referenced. * Syntax:

sql
    ALTER TABLE [catalog.][database.]table_name
    EXECUTE expire_snapshots(
        [ [older_than =] '<datetime>' ] [, [retain_last =] <int> ]
    );

* Default Behavior: If parameters are omitted, it typically removes snapshots older than 5 days and retains the most recent one. 2.

remove_orphan_files

This command is supported to clean up files that are not referenced by any valid snapshot metadata. * Syntax:

sql
    ALTER TABLE [catalog.][database.]table_name
    EXECUTE remove_orphan_files(
        [ [older_than =] '<datetime>' ] [, [location =] '<string>' ]
    );

* Default Behavior: Removes orphan files older than 7 days from the table location. 3.

rewrite_manifests

This command is supported to optimize metadata by merging small manifest files into larger ones to improve query planning performance. * Syntax:

sql
    ALTER TABLE [catalog.][database.]table_name
    EXECUTE rewrite_manifests();

* Note: This operation currently rewrites data manifests for the current snapshot only. Prerequisites * Catalog Type: You must be using an Iceberg Catalog (e.g., Hive, Glue, or Rest) with appropriate write permissions. * Version: While

rewrite_data_files

and

expire_snapshots

were introduced earlier (around v3.2), full support for

rewrite_manifests

and advanced parameters for the others was solidified in the v4.0/v4.1 release cycle. References • data_source/catalog/iceberg/procedures.md • data_source/catalog/iceberg/iceberg_meta_table.md • data_source/catalog/iceberg/iceberg_catalog.md

@Rocky what other iceberg table maintenance operations starrocks supports ?

In addition to

rewrite_manifests

,

expire_snapshots

, and

remove_orphan_files

, StarRocks supports several other Iceberg table maintenance and management operations via the

ALTER TABLE ... EXECUTE

syntax and standard DDL: 1. Data File Compaction (

rewrite_data_files

) Used to merge small data files into larger ones to optimize query performance and reduce metadata overhead. * Syntax:

sql
    ALTER TABLE table_name EXECUTE rewrite_data_files("key"=value, ...);

* Key Parameters: You can specify

min_file_size_bytes

(default 256MB) and use a

WHERE

clause to target specific partitions. 2. Snapshot Management These commands allow you to manage the table's state and history: *

rollback_to_snapshot(<snapshot_id>)

: Reverts the table to a previous state using a specific snapshot ID. *

cherrypick_snapshot(<snapshot_id>)

: Applies the changes from a specific snapshot to the current table state without necessarily rolling back everything else. 3. Data Loading & Registration *

add_files

: Directly adds existing Parquet or ORC data files from a storage location (like S3) or another table into the Iceberg table metadata without rewriting the data. *

register_table

: (Used via

CALL system.register_table(...)

) Allows you to register an existing Iceberg table into your StarRocks catalog by pointing to its

metadata.json

file. 4. Branching and Tagging (v3.4+) StarRocks supports Iceberg's snapshot branching and tagging for advanced lifecycle management: *

CREATE BRANCH / DROP BRANCH

: Create independent lineages for testing or auditing. *

CREATE TAG / DROP TAG

: Mark specific snapshots (e.g., "yearly-backup-2023") for long-term retention. * Snapshot Retention: You can define retention policies (number of snapshots or age) specifically for these branches using the

WITH SNAPSHOT RETENTION

clause. 5. Partition Evolution Unlike traditional systems, StarRocks supports Iceberg's In-place Partition Evolution, allowing you to update the partitioning scheme without rewriting the table: *

ALTER TABLE table_name ADD PARTITION COLUMN ...

*

ALTER TABLE table_name DROP PARTITION COLUMN ...

6. Metadata Inspection (v3.4.1+) While not a "command" that changes data, you can perform maintenance audits by querying Iceberg Metadata Tables: _

SELECT _ FROM table_name$snapshots;

_

SELECT _ FROM table_name$history;

_

SELECT _ FROM table_name$manifests;

_

SELECT _ FROM table_name$files;

References • data_source/catalog/iceberg/procedures.md • data_source/catalog/iceberg/iceberg_timetravel.md • data_source/catalog/iceberg/iceberg_meta_table.md References • data_source/catalog/iceberg/procedures.md • data_source/catalog/iceberg/iceberg_timetravel.md • administration/management/Backup_and_restore.md • data_source/catalog/iceberg/iceberg_meta_table.md