@Rocky is there any retention policy for cluster snapshots? or I should manually manage to delete older snapshots in object store?

The answer depends on whether you are referring to the Cluster Snapshot feature for shared-data clusters or the standard BACKUP (snapshot) functionality. 1. Cluster Snapshot (Shared-data Clusters only) If you are using the automated Cluster Snapshot feature (introduced in v3.4.2 for disaster recovery): * Retention Policy: There is an automatic retention policy, but it is strict: only one snapshot is retained. * Automatic Management: The system automatically maintains a snapshot that follows the latest cluster state. When a new automated snapshot is created, the previous historical snapshot is automatically dropped by the system. * Manual Deletion: You do not need to manually manage these in the object store; StarRocks handles the cleanup of the previous automated snapshot. If you disable the feature (

ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF

), the system automatically purges the existing snapshot. 2. Standard BACKUP Snapshots (Shared-nothing Clusters) If you are using the standard

BACKUP SNAPSHOT ... TO <repository>

command to back up data to a remote repository (S3, HDFS, etc.): * Retention Policy: There is no automatic retention policy. StarRocks will keep every snapshot you create in the repository indefinitely. * Manual Management Required: StarRocks does not provide a SQL command (like

DROP SNAPSHOT

) to delete data from the remote repository. * How to delete: You must manually delete the snapshot files directly in your object store (e.g., using the S3 console or CLI) to release space. * Note: Using

DROP REPOSITORY

only removes the StarRocks metadata/mapping for that storage location; it does not delete the actual backup data in the object store. Summary Table Feature | Cluster Snapshot (DR) | Standard BACKUP Supported Cluster | Shared-data only | Shared-nothing Automatic Cleanup | Yes (keeps only the latest 1) | No Manual Action | Not needed | Required (Delete in Object Store) StarRocks Version | v3.4.2+ | All Recommendation: If you are performing regular backups using the

BACKUP

command, you should set up a lifecycle policy in your object storage (like AWS S3 Lifecycle Rules) or a scheduled script to delete folders older than your desired retention period. References • administration/management/Backup_and_restore.md • faq/Exporting_faq.md

@Rocky why I can have 1tb of data in shared cluster in object store fe/meta?

In a shared-data StarRocks cluster, seeing 1TB of data in an

fe/meta

or similar metadata directory in object storage is usually tied to one of two things: the Automated Cluster Snapshot feature or the scale of your Tablet Metadata. 1. Automated Cluster Snapshot (FE Metadata) If you have enabled the Automated Cluster Snapshot feature (introduced in v3.4.2), StarRocks periodically uploads the FE's internal state (BDBJE images and edit logs) to your object store. * Location: These are stored under

/{storage_volume_path}/{service_id}/meta/image/automated_cluster_snapshot_timestamp

. * Retention: By default, StarRocks follows a strict "keep only one" policy. When a new automated snapshot is created, the system is supposed to drop the previous one. * Why it might be 1TB: * Scale: If your cluster has an extremely large number of objects (millions of tablets, thousands of users/privileges), a single metadata image can be massive. * Cleanup Issues: If the automated cleanup fails or if you have created several manual cluster snapshots (

ADMIN SET CLUSTER SNAPSHOT ...

), they will persist in the object store until you manually delete them. * Checkpoints: The snapshot is triggered every time the FE completes a metadata checkpoint (controlled by

automated_cluster_snapshot_interval_seconds

, default 10 minutes). 2. Tablet Metadata (The most likely culprit) In shared-data mode, StarRocks stores the "manifest" for every data file (segment) in the object store. This is called Tablet Metadata. * Structure: Since v3.1.4, each partition in the object store has a

meta/

subdirectory. * Why it grows: Unlike your actual data files (which are compressed and large), tablet metadata consists of many small files describing which segments belong to which tablet. * If you have over-partitioned your data or have an excessive number of buckets, you will generate millions of small metadata files. * In very large clusters, the cumulative size of these small metadata files can reach 1TB, especially if there is high data churn (frequent loads and compactions). * Retention: This metadata is not part of a "policy"—it is the live definition of your data. It is only deleted when: 1. A table or partition is dropped (using

DROP ... FORCE

or after the

catalog_trash_expire_second

period). 2. Compaction merges old versions, and the background Garbage Collection (GC) process cleans up the obsolete metadata files. Summary & Action * If you see many folders named

automated_cluster_snapshot_...

: Your cluster is failing to purge old snapshots. You can manually delete the older folders in the object store or check

information_schema.cluster_snapshots

to see what the FE thinks exists. * If the space is taken by millions of small files in table directories: You likely have a "small file" metadata problem. You should consider: * Reducing the number of partitions or buckets (aim for larger tablets, e.g., 1GB-10GB). * Checking if your Garbage Collection (GC) is keeping up with data churn. * Verification: Run

SHOW PROC '/dbs/<db_name>/<table_name>/partitions';

to see the

DataSize

vs. the actual metadata overhead. References • administration/cluster_snapshot.md • faq/shared_data_faq.md • administration/management/FE_configuration.md References • administration/cluster_snapshot.md • administration/management/FE_configuration.md

@Rocky Like I said I use shared arhcitechture and what can be the problem of too many small files in fe/meta object store?

In a StarRocks shared-data architecture, finding 1TB of data—especially if composed of many small files—in the

fe/meta

directory of your object store is usually a symptom of an issue with the Automated Cluster Snapshot feature or a misconfiguration of metadata management. Here are the specific problems caused by having too many small files in the

fe/meta

object store: 1. High API Costs (Object Storage Charges) Object storage providers (like AWS S3) charge per request (PUT, LIST, GET). * PUT/LIST overhead: Every time a snapshot is created (default every 10 minutes), the FE performs many PUT operations. If the cleanup process fails, millions of small metadata files accumulate. * Listing Costs: Commands that check for existing snapshots or background cleanup tasks must perform

LIST

operations. If there are millions of files, these LIST operations become extremely expensive and slow. 2. Significant Increase in RTO (Recovery Time Objective) The primary purpose of these snapshots is Disaster Recovery. * If you need to restore your cluster, the FE must download these metadata files from the object store. * Downloading 1TB of millions of small files is exponentially slower than downloading a few large files due to the overhead of establishing a connection for each file. This can turn a 10-minute recovery into a multi-hour outage. 3. Metadata Cleanup Failure (Garbage Collection) StarRocks' background threads are responsible for purging old snapshots. * Prefix Limits: Object stores have limits on how many requests per second can be made to a specific prefix (e.g., S3's 3,500 PUTs/5,500 GETs per second). If you hit these limits due to "small file bloat," the cleanup thread may fail to delete old files, causing the 1TB volume to grow indefinitely. * Memory Pressure: The FE must track these snapshots. An excessive number of un-purged snapshots can increase the memory overhead on the Leader FE. 4. Slow FE Performance The Leader FE manages the synchronization of these files. If the object store becomes sluggish due to the sheer volume of files in a single prefix, it can cause the FE's checkpointing process to lag, which may indirectly impact cluster stability and the synchronization of metadata to Follower FEs. ────────── Why is it 1TB? (Root Causes to Check) FE metadata (images and logs) is usually quite small (MBs to several GBs). For it to reach 1TB, one of the following is likely happening: 1. Retention Failure: The system is failing to drop the "historical snapshots" as intended. Check

SELECT * FROM information_schema.cluster_snapshots;

. If you see a long list of snapshots, they are not being deleted. 2. Snapshot Frequency: Your

automated_cluster_snapshot_interval_seconds

might be set too low, creating snapshots faster than they can be cleaned up. 3. Large Tablet Metadata: If you have millions of tablets/buckets, each FE "image" becomes massive. StarRocks snapshots include the state of all tables. 4. Manual Snapshots: If users have been running

ADMIN SET CLUSTER SNAPSHOT ...

manually without deleting old ones, they will persist forever. Recommendation 1. Check Snapshots: Run

SELECT * FROM information_schema.cluster_snapshots;

to see how many exist. 2. Verify Version: Ensure you are on a version with stable cleanup (v3.2.x or v3.3.x+). 3. Manual Cleanup: If there are thousands of folders in

fe/meta/image/

that are not listed in the

information_schema.cluster_snapshots

table, you may need to manually delete those directories in S3 to stop the "small file" performance drain. 4. Adjust Interval: If your metadata is large, increase

automated_cluster_snapshot_interval_seconds

(e.g., to 3600 for hourly) to reduce the churn of small files. References • administration/cluster_snapshot.md • faq/shared_data_faq.md References

@Rocky in shared cluster in object store in meta/fe I have big files and there are meta/data and schema files in it what it can be?

In a StarRocks shared-data cluster, seeing a

meta/fe

(or

fe/meta

) directory in your object store containing files like

meta/data

and

schema

is expected. This directory represents the persistent state of the FE's internal metadata service (often referred to as the Lake Metadata or StarMgr state). 1. What are these files? Unlike the actual data segments (stored in table-specific paths), the files in

meta/fe

are the "brains" of your shared-data cluster. They allow the FE to be "stateless" or recoverable from object storage. *

schema

: This file contains the schema definitions for the internal metadata records. It defines how the FE interprets the tablet-to-object mappings, transaction logs, and cluster state. *

meta/data

: These are the serialized data files (often in a key-value or log format) that store the actual metadata content. This includes: * Tablet Mapping: Which tablets belong to which tables. * Version History: Which data files (segments) in S3 belong to which version of a tablet. * Task State: Information about ongoing loads and compaction tasks. 2. Why are they "Big"? While 1TB of metadata is quite large, its size scales with the complexity of your cluster: * High Tablet Count: If you have millions of tablets (due to high bucket counts or excessive partitioning), the metadata describing those tablets grows significantly. * Data Churn: Frequent loads and updates create many "versions" of tablet metadata. If the Vacuum process (garbage collection) is not keeping up, historical versions of this metadata will accumulate. * Automated Snapshots: If you have enabled Automated Cluster Snapshots (v3.4.2+), the FE periodically dumps its entire internal BDBJE state to this location for disaster recovery. These snapshots can be several GBs each, and if multiple are retained, they add up. 3. Is it a problem? * Normally, no: This is required for the cluster to function. If you delete these files, the FE will lose its mapping to the data in the object store, effectively corrupting the cluster. * Performance concern: If the number of small files in this directory becomes excessive (millions of files), it can increase S3 API costs and slow down FE startup/recovery times. Recommendations 1. Check Retention: Run

SELECT * FROM information_schema.cluster_snapshots;

to see if old snapshots are being kept. 2. Monitor Vacuum: Ensure your

lake_autovacuum_grace_period_minutes

(default is usually 24 hours) is set appropriately. This determines how long old metadata versions are kept before being purged. 3. Audit Tablet Count: If your metadata is disproportionately large, check if you have too many tablets. Aim for tablet sizes of 100MB–1GB to keep the metadata volume manageable. Warning: Do not manually delete or move files in the

meta/fe

prefix unless instructed by support, as this directory contains the global state required for the cluster to identify and read data to your. needed to link your SQL tables to the physical files in storage. References • administration/management/FE_configuration.md • administration/Meta_recovery.md • introduction/Architecture.md