@Rocky can I run compaction/vacuum in starrocks shared cluster on object store iceberg tables?

Yes, you can perform compaction and vacuum-like operations on Iceberg tables in StarRocks. In StarRocks, these tasks are handled via Iceberg Stored Procedures using the

ALTER TABLE ... EXECUTE

syntax. This applies to Iceberg tables accessed via an Iceberg Catalog, regardless of whether your StarRocks cluster is in a "shared-data" (Compute-Storage separation) or "shared-nothing" architecture. 1. Compaction (Merging Small Files) To perform compaction (rewriting data files to merge small ones and optimize layout), use the

rewrite_data_files

procedure:

sql
ALTER TABLE [catalog.][database.]table_name 
EXECUTE rewrite_data_files(
    "min_file_size_bytes" = 134217728 -- Example: 128MB
) 
[WHERE <predicate>];

* Manual Trigger: This is a manual operation. You can specify a

WHERE

clause to target specific partitions. * Shared-Data Note: In a shared-data (CN-based) cluster, the Compute Nodes (CNs) execute these rewrite tasks. 2. Vacuum (Cleaning Up Data) Iceberg "vacuuming" is split into two main procedures in StarRocks: * Expire Snapshots: Deletes data files associated with old snapshots that are no longer needed.

sql
    ALTER TABLE iceberg_catalog.db_name.table_name
    EXECUTE expire_snapshots(older_than = '2024-01-01 00:00:00', retain_last = 2);

* Remove Orphan Files: Removes files in the table's storage location that are not referenced by any valid snapshot metadata.

sql
    ALTER TABLE iceberg_catalog.db_name.table_name
    EXECUTE remove_orphan_files(older_than = '2024-01-01 00:00:00');

3. Metadata Optimization You can also optimize the manifest files to improve query planning performance:

sql
ALTER TABLE iceberg_catalog.db_name.table_name EXECUTE rewrite_manifests();

Key Differences to Note * StarRocks Native Tables: If you are using StarRocks' internal shared-data tables (storing StarRocks' own segment format on S3/OSS), compaction is automatic and controlled by the FE. * Iceberg Tables: For tables in an Iceberg Catalog, StarRocks currently requires you to trigger these maintenance procedures manually (or via a scheduled task/airflow). For more details, you can refer to the official Iceberg Stored Procedures documentation. References • data_source/catalog/iceberg/procedures.md • administration/management/compaction.md • data_source/catalog/iceberg/iceberg_timetravel.md