We have about 45 tables in a starrocks shared-data cluster backed by S3, this is a fairly small data set, ~1GB. We offload most of this dataset into another bucket as compressed parquet files summing up to ~340MB, however the bucket used for starrocks data is currently somehow grown to 37TB

@Rocky - what can we do to find out why a shared data cluster on s3 has a much larger starrocks bucket now. Is there any space optimization or investigation that can help?

To find out why your StarRocks shared-data cluster on S3 has grown significantly and how to optimize it, you can follow these investigation and optimization steps: 1. Investigation: Identify the "Storage Hogs" First, determine which tables or partitions are consuming the most space and whether compaction is keeping up. * Check Partition Metadata: Query the

information_schema.partitions_meta

table to identify partitions with a large

DATA_SIZE

or a high

MAX_CS

(Compaction Score). A high compaction score means there are many small files/versions that haven't been merged yet.

sql
    SELECT DB_NAME, TABLE_NAME, PARTITION_NAME, DATA_SIZE, MAX_CS
    FROM information_schema.partitions_meta
    ORDER BY DATA_SIZE DESC;

* Check Compaction Status: If the compaction score is high, check if compaction tasks are actually running or failing:

sql
    SHOW PROC '/compactions';
    -- Or for more detail:
    SELECT * FROM information_schema.be_cloud_native_compactions;

* Identify Storage Paths: You can see exactly where each table is stored in S3 using:

sql
    SHOW PROC "/dbs/<db_id>/<table_id>/partitions";

2. Optimization: Adjust Data Retention (AutoVacuum) In shared-data mode, StarRocks doesn't delete S3 files immediately after a new version is created (via compaction or ingestion) to ensure that ongoing queries can still access the old data. *

lake_autovacuum_grace_period_minutes

(FE Config): This parameter controls how long historical data versions are retained. The default is often 30 minutes (in newer versions). If this was accidentally set to a very high value, your S3 bucket will grow because old versions aren't being cleaned up. * To check:

ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum_grace_period_minutes';

* To adjust:

ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "30");

*

lake_autovacuum_parallel_partitions

(FE Config): If you have thousands of partitions, the default cleanup parallelism (8) might be too slow. You can increase this to speed up the deletion of stale files. 3. Investigation: Garbage Collection (Failed Tasks) If your cluster is on an older version (specifically older than 3.1.8 or 3.2.3), certain failed ingestion or compaction tasks might leave "garbage" files in S3 that the system cannot automatically clean up. * Use the

meta_tool

for manual GC: StarRocks provides a tool to find and delete these orphaned files. You can run the

lake_datafile_gc

operation. Warning: Always run with

--do_delete=false

first to see what would be deleted.

bash
    ./bin/meta_tool.sh --operation=lake_datafile_gc \
      --root_path="<s3://your-bucket/your-cluster-id/>" \
      --expired_sec=86400 \
      --conf_file=./conf/be.conf \
      --do_delete=false

4. Compaction Tuning If you have a very high volume of small writes, compaction might be lagging, leading to many small files in S3. *

max_cumulative_compaction_num_singleton_deltas

(CN Config): It is recommended to set this to

100

so that each compaction task finishes faster and consumes fewer resources. *

compact_threads

(CN Config): Increase this if your Compute Nodes have spare CPU and the compaction queue is full. Summary Checklist 1. Lower

lake_autovacuum_grace_period_minutes

if it's set too high. 2. Verify Compaction is healthy (Score < 100). 3. Run

meta_tool.sh

if you suspect leaked files from failed jobs (especially on older versions). 4. Check for Data Skew in

information_schema.partitions_meta

. References • knowledge/admin_and_operation/shared_data_compaction.md • faq/resource_isolation_faq.md