@Rocky We are using S3 to store our data. I've enabled filter metrics and metadata tables, and I can see a very high load of DELETE into the S3, although the starrocks cluster is not doing much. The DELETEs also often aim the same files, which creates a tombstone each time, cluttering the versions of that file. My entire data fits in ~50GB, and I see 200k+ DELETE operations per day, most of them deleting the same files. I have added regular snapshots (every 8h) to starrocks. Do you know how to investigate this?

This behavior is typically seen in StarRocks Shared-data (compute-storage separated) clusters where metadata management and automatic background tasks interact with S3's versioning features. Probable Causes 1. S3 Bucket Versioning Interaction: StarRocks is designed to manage its own "data versions" and expects objects to be physically removed upon a

DELETE

request. If S3 Bucket Versioning is enabled, a

DeleteObject

request does not remove the file; it adds a Delete Marker (tombstone). If StarRocks' background garbage collection (GC) or compaction logic repeatedly attempts to "clean up" what it still perceives as a residual file (or simply executes its cleanup cycle), it will create a new Delete Marker version each time, leading to the clutter you see. 2. Cluster Snapshots & Metadata Rotation: You mentioned snapshots every 8 hours. In shared-data mode, the Cluster Snapshot feature maintains a consistent state of the cluster in S3. * StarRocks maintains only the latest snapshot; old ones are "dropped." * While a snapshot is active, it blocks garbage collection of the data versions it references to ensure restorability. * When a new snapshot is created and the old one is released, a large backlog of "deletable" files (from previous compactions or loads) may be processed at once, causing a spike in

DELETE

operations. 3. Primary Key Persistent Index (Shared-data only): If you are using Primary Key tables with persistent indexes enabled, StarRocks stores index files (SSTables) in S3. These files are frequently compacted and updated. High ingestion or even minor background updates can result in many small index files being created and then deleted shortly after. 4. Metadata File Updates: StarRocks stores "Tablet Meta" files in S3. Every time a tablet is updated (compaction, load, schema change), a new meta file is written and the old one is scheduled for deletion. With a high bucket/tablet count, this adds up quickly. ────────── Recommended Investigation Steps 1. Identify the "Target" Files Check your S3 Server Access Logs or CloudTrail to identify the specific prefixes of the files being deleted: *

.../meta/

: These are tablet metadata files. High volume here suggests frequent version updates or many tablets. *

.../data/

: These are actual data segments. High volume here indicates frequent compactions. *

.../persistent_index/

: (If applicable) This confirms the Primary Key index is the source. 2. Verify S3 Versioning Configuration If versioning is enabled on your bucket, consider the following: * Disable versioning if you rely on StarRocks' built-in snapshot/backup features for recovery. * If you must keep it enabled, configure an S3 Lifecycle Policy to "Permanently delete non-current versions" and "Delete expired object delete markers" after 1–7 days to prevent tombstone bloat. 3. Check Tablet Count and Compaction A small dataset (50GB) with 200k deletes suggests your data might be fragmented into too many tablets or buckets. * Run

SHOW FRAGMENTS FROM <table_name>;

to check the number of tablets. * Check the compaction score:

SHOW PROC '/dbs/<db_id>/<table_id>/partitions';

. If

MaxCS

is high, the system is struggling to merge files. 4. Audit Snapshot Impact Try temporarily disabling the 8-hour snapshots. If the

DELETE

volume drops significantly, the issue is likely the snapshot rotation logic or the release of "blocked" garbage data after each snapshot cycle. 5. Configuration Tuning (FE/BE) Check these parameters in your FE/BE configs: * FE:

catalog_trash_expire_second

(Default is usually 86400s / 24h). This controls how long metadata stays in the "recycle bin" before the physical

DELETE

is sent to S3. * BE:

trash_file_expire_time_sec

(Default 24h).

@Rocky I need to keep versioning activated because this enables AWS Backup for that S3. I manually added the 1 day expiration of tombstones. Regarding the target files, that's mainly "meta" files. How would we reduce this clutter ? 200k req / day is about 30 DELETES per seconds, that's not a normal rate. We update the data about once every hours, with about 1% of records update (so maybe 500MB). Also, it removes MANY TIMES the same file (most meta files are "removed" 400+ times a day), why doesn't it consider it "done" and stops removing it?

In StarRocks Shared-data (compute-storage separated) clusters, the behavior you're describing—where the same "meta" files are deleted hundreds of times a day—is usually a conflict between StarRocks' internal Vacuum logic and S3 Bucket Versioning. Why the "Same File" is Deleted Repeatedly The root cause is a logic loop triggered by how StarRocks identifies garbage: 1. Vacuum Scan: StarRocks runs a background "Vacuum" task (typically every few minutes). This task lists files in your S3 bucket to compare what is physically stored against what its internal metadata says is "active." 2. S3 Versioning Conflict: When StarRocks lists the bucket, if it uses an API call that returns all versions (or if the S3-compatible layer it uses doesn't ignore non-current versions), it sees the old file version still exists. 3. The Delete Loop: _ StarRocks thinks: _"I see file

meta/123_v1.meta

in S3, but I no longer need it. Send a

DELETE

request."* * S3 receives the

DELETE

. Because Versioning is enabled, S3 does not remove the file; it adds a Delete Marker (tombstone). _ A few minutes later, the Vacuum task runs again. It lists S3 and _still sees* the old versions of that file. _ StarRocks thinks: _"That file is still there! Delete it again."* _ S3 adds _another* Delete Marker. 4. Result: This repeats every few minutes (the Vacuum interval), creating 400+ Delete Markers per day for the same file key, bloating your S3 metadata and increasing your API costs/latency. How to Investigate & Fix 1. Adjust the Vacuum/GC Intervals You can reduce the frequency of these checks so they don't hit S3 30 times a second. Check these FE configurations: * `lake_autovacuum_interval_seconds`: (If available in your version) Increase this to reduce the frequency of the Vacuum loop. * `lake_autovacuum_grace_period_minutes`: This is the safety buffer. If a file was just "unreferenced," StarRocks waits this long before deleting it. Increasing this can batch deletions better. * `lake_vacuum_batch_size`: (Default 1000). Increasing this (up to 1000) makes each S3 call more efficient by deleting more files per request. 2. Configure S3 Lifecycle to Clean "Delete Markers" Since you must keep versioning for AWS Backup, you must ensure S3 itself cleans up the mess StarRocks is making. Ensure your Lifecycle Policy has: * "Delete expired object delete markers": This is critical. When an object has only a delete marker and no other versions, S3 can remove the marker entirely. * "Permanently delete non-current versions": Set this to 1 day. This ensures that once StarRocks "deletes" a file (adding a marker), the actual data version is purged by S3 24 hours later. 3. Tablet & Metadata Churn If your 50GB of data is split into too many tablets, every 1-hour update will generate a massive wave of new

.meta

files. * Check your tablet count:

SHOW FRAGMENTS FROM your_table;

* If you have thousands of tablets for 50GB, you have over-partitioned or over-bucketed. Aim for tablet sizes between 100MB and 1GB. Reducing the tablet count will linearly reduce the number of "meta" files created and deleted. 4. The "Snapshot" Interaction Your 8-hour snapshots are likely making this worse. * When a snapshot is active, it "pins" all current meta/data files. StarRocks cannot delete them. * When the snapshot expires every 8 hours, a massive queue of "to-be-deleted" files is released. The Vacuum process then works overtime to clear them, potentially hitting S3 rate limits. Recommended Actions 1. Reduce Vacuum Frequency: Increase FE

lake_autovacuum_interval_seconds

or related GC timers to stop the 3-minute loop. 2. Verify S3 Lifecycle: Double-check that

ExpiredObjectDeleteMarker

is set to

true

in your AWS S3 Lifecycle rule. 3. Optimize Bucketing: If you have many small tablets, re-create the table with fewer buckets to reduce the number of metadata files StarRocks has to track.