@Rocky I have a question about storage space in my S3 bucket. The starrocks cluster is the sole user, in shared_data mode. Versioning is activated, trash of starrocks is set to 30 days. My databases seem to be taking about 7GB:

select SUM(DATA_LENGTH) from information_schema.tables group by table_schema;
+------------------+
| sum(DATA_LENGTH) |
+------------------+
|       6637490216 |
|             NULL |
|         16060799 |
|         63917860 |
|             NULL |
+------------------+

But my bucket is reaching 160GB of storage and climbing (see screenshot). What can explain this ? How can I investigate?

The significant discrepancy between your StarRocks table size (~7GB) and your S3 bucket size (~160GB) in shared-data mode is usually caused by one of three things: S3 Versioning behavior, StarRocks historical version retention, or incomplete S3 uploads. 1. S3 Versioning (Primary Suspect) Since you have S3 Versioning activated, StarRocks' cleanup process behaves differently than you might expect: * The Problem: When StarRocks performs a "delete" (during Vacuum/Garbage Collection), S3 does not actually delete the object. Instead, it creates a Delete Marker and moves the old file to a Noncurrent Version. * The Result: Because StarRocks is an LSM-tree-based system, it creates and deletes thousands of small files during frequent ingestions and compactions. If your S3 bucket is not configured with a Lifecycle Policy to "Permanently delete noncurrent versions" after X days, your bucket size will grow indefinitely. * Investigation: Check your S3 bucket using S3 Storage Lens or S3 Inventory. Look for the "Noncurrent version" storage class. 2. StarRocks "Trash" and Vacuum Period You mentioned the "trash is set to 30 days." In shared-data mode, there are two distinct retention mechanisms: *

lake_autovacuum_grace_period_minutes

: This is the most critical parameter for shared-data storage. It controls how long StarRocks keeps historical data versions after they are no longer needed (e.g., after a compaction has merged them into a new file). * The Default: 30 minutes. * Your Setup: If you have set this to 30 days (43,200 minutes), StarRocks will keep every single version of every file created during the last month. For a high-churn cluster, this can easily result in 20x–50x storage inflation. *

catalog_trash_expire_second

: This controls how long dropped tables are kept in the FE recycle bin. This only applies to the metadata and data of tables you explicitly deleted with

DROP TABLE

. 3. Lagging Compaction or Vacuum If StarRocks cannot keep up with background tasks, old data files won't be marked for deletion: * Compaction: Check if compaction is falling behind.

sql
    -- Check MaxCS (Compaction Score). Anything over 100 indicates a backlog.
    SELECT DB_NAME, TABLE_NAME, MAX_CS FROM information_schema.partitions_meta;

* Vacuum Tasks: If the vacuum queue is stuck, files stay in S3 forever.

sql
    -- Check if tasks are running or backlogged
    SHOW PROC '/compactions';

4. Incomplete Multipart Uploads If ingestion tasks (like Broker Load or Stream Load) fail or are interrupted, S3 may store "Incomplete Multipart Uploads." These are "ghost" files that take up space but do not appear as regular objects in the S3 console. * Solution: Add an S3 Lifecycle Rule to "Abort incomplete multipart uploads" after 1 or 2 days. How to Investigate 1. Check FE Config: Run

ADMIN SHOW FRONTEND CONFIG LIKE '%lake_autovacuum%';

and check the value. If it is set to a high value, reduce it (the recommended default is 30–60 minutes). 2. Analyze S3 Storage: In the AWS S3 Console, use Storage Lens to see the breakdown of current vs. noncurrent versions. 3. Manual GC (For old versions): If you are on a StarRocks version earlier than 3.1.8 or 3.2.3, there was a known issue where failed compactions could leave "garbage" files that were never cleaned up. StarRocks provides a

lake_datafile_gc

tool specifically for this. Recommendation: Check your

lake_autovacuum_grace_period_minutes

first. If that is low, the culprit is almost certainly S3 Versioning without a matching S3 Lifecycle policy to purge noncurrent versions. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md