Hi @Rocky, Is there a way to automate the compaction of iceberg tables in starrocks?

While StarRocks does not currently provide a single "toggle" in the catalog properties for fully background automatic compaction of Iceberg tables, you can automate this process using StarRocks' built-in maintenance procedures combined with its Task scheduling system. 1. The Core Maintenance Procedures StarRocks provides several procedures to manage and compact Iceberg tables. The most relevant for compaction is

rewrite_data_files

, which merges small files into larger ones. * Compaction (Rewrite Data Files):

sql
    ALTER TABLE <iceberg_table> EXECUTE rewrite_data_files("min_file_size_bytes"="134217728");

* Snapshot Expiration: (Cleans up old data versions)

sql
    ALTER TABLE <iceberg_table> EXECUTE expire_snapshots(older_than = '2024-01-01 00:00:00', retain_last = 10);

* Orphan File Removal: (Cleans up unreferenced files)

sql
    ALTER TABLE <iceberg_table> EXECUTE remove_orphan_files();

2. How to Automate You can automate these by wrapping them in a StarRocks Scheduled Task (supported from v3.3+). This allows StarRocks to handle the execution logic without needing an external tool like Airflow. Example: Automating daily compaction

sql
CREATE TASK iceberg_compaction_daily
SCHEDULE EVERY(INTERVAL 1 DAY)
AS 
ALTER TABLE iceberg_catalog.db.my_table 
EXECUTE rewrite_data_files("min_file_size_bytes"="268435456");

3. Key Considerations * Resource Management: Compaction is resource-intensive. You can control the parallelism with the

batch_parallelism

property (e.g.,

EXECUTE rewrite_data_files("batch_parallelism"="4")

). * StarRocks 4.0+: In the latest versions (v4.0 and beyond), StarRocks has introduced a more integrated Compaction API for Iceberg designed to handle these tasks more natively and reduce the need for separate maintenance jobs. * External Scheduling: If your version of StarRocks does not support

ALTER TABLE

inside

CREATE TASK

, the standard approach is to use an external orchestrator (Cron, Airflow, or dbt) to send the

ALTER TABLE ... EXECUTE

command to_procedure` command to StarRocks on a schedule. For detailed syntax on parameters like

min_file_size_bytes

or

batch_size

, you can refer to the Iceberg Procedures documentation. References • data_source/catalog/iceberg/procedures.md • data_source/catalog/iceberg/iceberg_catalog.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md

Noted @Rocky, one more thing, can we deploy decoupled starrocks in kubernetes?

Yes, you can deploy "decoupled" StarRocks (officially called Shared-data architecture) on Kubernetes. In this mode, storage is separated from compute, allowing for elastic scaling and lower costs. 1. Recommended Deployment Method The best way to do this is using the StarRocks Kubernetes Operator. It handles the orchestration of the shared-data components natively. * Compute Nodes (CN): In the decoupled architecture, the traditional Backend (BE) nodes are replaced by stateless CN nodes. These nodes handle the computation and cache hot data locally. * Object Storage: Data is stored in a central shared storage like AWS S3, Google GCS, Azure Blob Storage, or MinIO. 2. Key Benefits on Kubernetes * Elastic Scaling: Since CNs are stateless (data is in S3), you can scale them up or down in seconds without waiting for data migration or rebalancing. * Auto-scaling: You can use the Kubernetes Horizontal Pod Autoscaler (HPA) to automatically scale your compute power based on CPU or memory usage. * Resource Isolation: You can deploy different CN groups (warehouses) for different workloads (e.g., one for ETL, one for BI) all pointing to the same shared storage. 3. High-Level Configuration To enable this mode, you typically configure the following in your StarRocks cluster custom resource or configuration files: * FE Configuration (

fe.conf

):

properties
    run_mode = shared_data
    cloud_native_storage_type = S3  # or Azure/HDFS

* Storage Volume: You define a "Storage Volume" in StarRocks that points to your S3/MinIO bucket. 4. Implementation You can find the StarRocks Operator and deployment examples on the StarRocks GitHub. The documentation specifically covers the

StarRocksCluster

CRD (Custom Resource Definition) which you would use to define your shared-data cluster. References • deployment/shared_data/feature-support-shared-data.md • quick_start/shared-data.md • deployment/shared_data/s3.md References • deployment/shared_data/feature-support-shared-data.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • deployment/shared_data/s3.md

Hi @Rocky Does the shared data mode use buckets at all? Because I can see some tablets are specified 4 buckets and when I run show tablets, I can see 4 buckets 23626 [23018] 30.7MB 23627 [23018] 30.7MB 23628 [10001] 30.8MB 23629 [10001] 30.7MB But when I go to the remote storage, I only see one bucket and 4 files packed into that bucket for this particular table

Yes, the shared-data (decoupled) mode still uses the concept of buckets (internally called tablets). Even though the storage is moved to remote object storage (like S3 or MinIO), StarRocks still uses tablets as the logical unit for data distribution and query parallelism. 1. Why you see 4 buckets in

SHOW TABLETS

When you define

DISTRIBUTED BY HASH(...) BUCKETS 4

in your DDL, StarRocks logically divides your data into 4 tablets. This ensures that: * Parallelism: Up to 4 compute threads/nodes can process this data in parallel. * Distribution: Data is distributed across these 4 units to prevent "hotspots." 2. Why you see 4 files in remote storage In the shared-data architecture, each tablet's data is stored as one or more segment files in your remote storage. If you see exactly 4 files in your object storage for a table with 4 buckets, it most likely means that each tablet has produced exactly one segment file (common after a load or a compaction process). 3. How they are organized In recent versions (v3.1.4+), StarRocks organizes these files in your S3/MinIO bucket using a specific path structure:

.../<table_id>/<partition_id>/data/<segment_id>.dat

* The

.dat

files are the actual data segments. * There are also

.meta

files (in a

meta/

subdirectory) that store the mapping between the Tablet ID and the physical segment files. 4. Verification To see exactly where a table is stored in your remote bucket, you can use:

sql
SHOW PROC '/dbs/<your_db_id>/<your_table_id>';

Or to get the IDs needed for that command:

sql
SHOW PROC '/dbs'; -- Get DB ID
SHOW PROC '/dbs/<db_id>'; -- Get Table ID

In summary: "Buckets" are the logical partitions of your data used for processing, and in your case, each of those 4 logical buckets has been written as 1 physical file in the remote storage. References • faq/shared_data_faq.md • table_design/table_design.md • deployment/shared_data/minio.md • deployment/shared_data/s3.md • deployment/shared_data/s3.md • deployment/shared_data/gcs.md • deployment/shared_data/azure.md • quick_start/shared-data.md References • deployment/shared_data/minio.md • deployment/shared_data/s3.md • quick_start/shared-data.md

Hi @Rocky, is it possible to restore the metadata of the FE of one starrocks cluster to a different starrocks deployment maybe as part of a DR approach?

Yes, it is possible to restore the FE metadata to a different cluster, and this is a standard manual approach for Disaster Recovery (DR) or cluster migration. The process involves moving the physical metadata directory (

meta_dir

) and forcing the new FE to become a leader without its previous peers. 1. The Manual Recovery Process If you have a backup of the

meta/

directory from your original cluster, you can initialize a new cluster with it by following these steps: 1. Prepare the new FE: Install StarRocks on the new environment, but do not start it yet. 2. Copy the metadata: Replace the

meta/

directory on the new FE with the backup from your original cluster. 3. Modify `fe.conf`: Add the following parameter to allow this FE to ignore the old cluster's election group and start as a new Leader:

properties
    bdbje_reset_election_group = true

4. Start the FE: Start the node using

./bin/start_fe.sh

. 5. Verify: Run

SHOW FRONTENDS;

. The node should show its role as

LEADER

. Check your tables and data to ensure the metadata is intact. 6. Finalize: Once verified, remove the

bdbje_reset_election_group

line from

fe.conf

and restart the FE normally. You can then add more FEs to this new cluster using the standard

--helper

method. 2. Special Considerations for Shared-Data (Decoupled) Mode Since you mentioned using decoupled (shared-data) StarRocks earlier, there are two critical things to remember for DR: * Remote Storage Access: The restored FE metadata contains the configuration for your "Storage Volumes" (e.g., S3/MinIO bucket names and credentials). The new DR environment must have network access to those same buckets or a replicated version of them. * Storage Consistency: If you are using cross-region replication for your S3/MinIO data, ensure that the metadata you restore is consistent with the version of the data in the DR bucket. If the metadata is "ahead" of the data replication, you may encounter errors. 3. Alternative: BACKUP and RESTORE Commands StarRocks also supports SQL-level

BACKUP

and

RESTORE

commands. These are typically used to back up specific databases or tables to a repository (S3/HDFS). * Pros: Cleaner, managed by StarRocks, avoids manual file copying. * Cons: Restores data/tables, but doesn't necessarily restore system-level settings or users as easily as a full

meta/

directory copy. Recommendation: For a full cluster-level DR where you want to keep all users, permissions, and configurations, the Metadata Recovery method (copying the

meta/

directory) is the preferred way to "clone" the FE state to a new deployment. References • administration/Meta_recovery.md • sql-reference/sql-statements/backup_restore/BACKUP.md • sql-reference/sql-statements/backup_restore/RESTORE.md References • administration/Meta_recovery.md