Hi, I'm wondering what are current best practices for deleting a StarRocks cluster while preserving the data in S3 in shared data mode. What I've figured out so far: • FE nodes store metadata (DBs, table definitions) in their PVCs (not S3) • If I do helm uninstall for the kube-starrocks chart and delete the PVCs using

persistentVolumeClaimRetentionPolicy

for the FE nodes the data (DBs and their tables do not exist) cannot be queried if I install the chart again connected to the same S3 bucket Rocky suggested that Cluster Snapshot is similar, it is however: • If that is the only option StarRock's shared data separated compute from data claim is not entirely true, because: ◦ One would assume on deleting the cluster and re-creating it using the same storage (s3) they'd be able to pick up where they left of and here this is not autoamtic • For shared data clusters it should not backup the data also as is stored in S3/elsewhere already • There's no FE config way to enable Cluster Snapshot for all storage volumes • There's no simple way to auto restore the cluster ◦ The word disaster recovery when it means restore is a bit scary, especially to a new user like me, this doc does explain how to do it which is nice though, but is manual and have to set a specific snapshot to be used In my opinion it would be more user friendly to have the following flow: 1. User enables cluster snapshot for metadata only for all storage volumes via a FE config 2. User enables automatic restore via a kube-starrocks config 3. User does helm install to spawn a new cluster, it backs up its FE metadata every X seconds 4. User does helm uninstall, deletes the PVCs 5. User does helm install using the same unchanged config, the cluster picks up the latest snapshot and restores it automatically without the user having to think about it An even neater approach would be: • Write the FE metadata to S3, then cache it locally and read it from there the same logic the CE nodes use ◦ This way the whole recovery process wouldn't be needed and can be used only for actual disaster recovery ◦ Should be considered how often, when/why is the metadata written to and how it would affect performance What do people think?

Hello! I'm Rocky, your StarRocks assistant. I'm here to help you get the most out of StarRocks. You can ask me questions about: * SQL Syntax & Usage: How to write queries, use window functions, or manage partitions. * Data Loading: Best practices for Routine Load, Stream Load, or using connectors. * Performance Tuning: How to optimize slow queries, use CBO, or configure indexes. * Architecture & Deployment: Questions about FE/BE configuration and cluster management. * Troubleshooting: Troubleshooting specific error messages or unexpected behavior. If you have a specific error or a "how-to" question, just paste it here! I'll search the official StarRocks documentation to provide you with the most accurate answer. What can I help you with today?

@Rocky I'm wondering what are current best practices for deleting a StarRocks cluster while preserving the data in S3 in shared data mode. What I've figured out so far: • FE nodes store metadata (DBs, table definitions) in their PVCs (not S3) • If I do helm uninstall for the kube-starrocks chart and delete the PVCs using

persistentVolumeClaimRetentionPolicy

for the FE nodes the data (DBs and their tables do not exist) cannot be queried if I install the chart again connected to the same S3 bucket If no better solution exists I'm proposing: • FE nodes store their metadata data in S3, using the same logic the CE nodes use cache it locally and read it from there. ◦ Things to consider here would be ▪︎ what exactly is FE metadata/how often is it written to • depending on this it might be better to do sync with S3 on graceful shutdown/boot instead of writing to S3 first then reading from local cache if it affects query latency

Hi! I'm Rocky. You've hit on a key architectural detail of StarRocks in shared-data mode. You are correct that while table data lives in S3, the metadata (the "brain" that knows which files belong to which table, database schemas, user permissions, etc.) traditionally resides in the FE's local

meta

directory (backed by PVCs in Kubernetes). As of today, here are the best practices for handling this scenario: 1. The Modern Solution: Cluster Snapshot (v3.4.2+) Your proposal—storing FE metadata in S3 to allow for cluster-wide recovery—is exactly what the StarRocks team has implemented with the Cluster Snapshot feature. Introduced in v3.4.2, Cluster Snapshot is designed specifically for disaster recovery and cluster migration in shared-data mode. It automatically ensures that the full cluster state (metadata + data) is persisted in object storage. * How it works: The FE periodically syncs a snapshot of its metadata to S3. * Why it helps you: If you delete your cluster (including FE PVCs), you can start a new cluster and point it to the snapshot in S3. The new Leader FE can bootstrap itself from that snapshot using the

--cluster_snapshot

startup flag. * Best Practice: Upgrade to v3.4.2+ and enable Cluster Snapshot. This removes the hard dependency on FE PVCs for long-term data preservation. 2. The Infrastructure Best Practice: FE PVC Retention If you are on an older version or simply want to keep the cluster "warm" between uninstalls, you should configure your Kubernetes environment to treat FE PVCs as "precious" state: * Retention Policy: Ensure your

persistentVolumeClaimRetentionPolicy

for the FE StatefulSet is set to

Retain

(or just don't delete them during

helm uninstall

). * Re-attachment: When you reinstall the Helm chart, ensure it uses the same

metadata.name

and volume claims so that the new FE pods mount the old metadata. _ _Note:* FE nodes must be started in a specific order if you are recovering from a completely stopped state to ensure the Leader (which has the latest edit logs) is elected correctly. 3. Alternative: Table-level BACKUP/RESTORE If you need to move data between entirely different S3 paths or different cluster versions where Cluster Snapshot isn't an option: * You can use the

BACKUP

command to create a snapshot of specific databases/tables into an S3 repository. * The

RESTORE

command can then be used in a fresh cluster to "import" that data. _ _Note:* This is more manual than Cluster Snapshot but is very robust for migration. Regarding your Proposal Your logic about caching and syncing metadata is very close to how the internal implementation of Cluster Snapshot and the FE's BDBJE (Berkeley DB Java Edition) handles things. The FE metadata is updated frequently (every transaction), so writing directly to S3 for every small metadata change would indeed introduce unacceptable latency for SQL DDL/DML. Instead, StarRocks uses a checkpointing mechanism: 1. FE writes metadata locally (high performance). 2. Periodic checkpoints/snapshots are taken. 3. These snapshots are then uploaded to S3. Summary Recommendation: If you want to be able to delete your cluster and delete PVCs while keeping everything queryable later, upgrade to v3.4.2+ and use the Cluster Snapshot feature. It was built specifically to solve the exact problem you've identified! References • administration/cluster_snapshot.md • administration/management/Backup_and_restore.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • deployment/shared_data/gcs.md

@Rocky what is stored in FE metadata

In StarRocks, the FE (Front End) metadata is the "source of truth" for the entire cluster. It stores all the logical and physical information required for the cluster to function, plan queries, and manage data. The FE metadata is primarily stored using BDB-JE (Berkeley DB Java Edition) and consists of two main types of files: image files (periodic snapshots) and edit logs (incremental logs of changes). Here is a breakdown of what is stored in the FE metadata: 1. Cluster Topology and Node Information * FE Nodes: Information about all FE nodes (Leader, Followers, Observers), including their IPs, ports, and roles. * BE/CN Nodes: A list of all Backend (BE) or Compute Nodes (CN), their heartbeat status, and disk usage. 2. Logical Metadata (Database & Table Schema) * Hierarchical Structure: Definitions of Databases, Tables, and Views. * Table Schema: Column names, data types, indexing information (e.g., Bitmap indexes, Bloom filter columns), and table properties. * Partition Information: Range or List partition definitions, partition keys, and individual partition states. 3. Physical Metadata (Tablet Distribution) * Tablet Mapping: Which BE nodes host which tablets (the physical data shards). * Replica Management: The location and status of each replica for a tablet. * Version Info: The "Visible Version" of each tablet/partition, which ensures consistency during queries. 4. Load and Transaction Information * Transaction Logs: Status of data ingestion transactions (Running, Committed, Visible, or Aborted). * Load Jobs: Metadata for various load types like Broker Load, Routine Load (e.g., Kafka offsets), and Stream Load jobs. * Export Jobs: Status and history of data export tasks. 5. Security and Permissions (RBAC) * Users & Roles: Authentication details for users and the roles assigned to them. * Privileges: Access control lists (ACLs) defining who can read, write, or manage specific databases and tables. 6. Background Task States * Dynamic Partitioning: Information for the dynamic creation and deletion of partitions based on time. * Materialized Views: Refresh states and definitions of asynchronous Materialized Views. * Compaction Scores: Internal metrics used by the FE to schedule and track data compaction on the BEs. How it is Stored The metadata is kept in memory by the FE for fast access during query planning. For persistence: * Edit Logs: Every change (like creating a table or finishing a load) is written to a journal (edit log) in BDB-JE. * Image Files: Periodically, the FE checkpoints the memory state into an

image

file and cleans up old edit logs to prevent them from growing indefinitely. * Location: By default, these files are located in the

${STARROCKS_HOME}/meta

directory (configurable via

meta_dir

). Note: Because the metadata is the "brain" of the cluster, it is highly recommended to deploy at least three FE Follower nodes to ensure high availability and metadata redundancy. References • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky how does Cluster Snapshot affect cluster performance?

The Cluster Snapshot feature (available from v3.4.2 in shared-data clusters) is designed to provide disaster recovery by backing up the entire cluster state (metadata and data) to object storage. Regarding its impact on cluster performance, here are the key aspects: 1. Minimal Impact on Query Performance Since StarRocks shared-data architecture already stores data in object storage, "creating" a data snapshot does not involve copying large amounts of data between nodes or to external systems. * Zero-copy for data: Data snapshots point to the same files already in object storage. There is no heavy I/O on the Compute Nodes (CNs) or Backend (BE) during the snapshot process. * Query Latency: Queries continue to run normally without significant resource contention from the snapshot process. 2. FE Metadata Overhead The primary performance impact is on the Frontend (FE) nodes: * Metadata Imaging: Each time the FE completes a metadata checkpoint (controlled by the snapshot interval, default 10 minutes), it must serialize its state and upload the image to object storage. * Resource Usage: During this phase, you may see a slight, temporary increase in CPU and network egress on the FE as it uploads the metadata image. 3. StarManager/Vacuum Impact * File Retention: Because snapshots preserve a specific state of the cluster, data files referenced by the current snapshot cannot be deleted by the vacuum (garbage collection) process. * Storage Usage: While this doesn't slow down the cluster, it can lead to a slight increase in object storage usage because historical data files are held until the next automated snapshot is successfully created and the old one is dropped. 4. Configuration Tuning If you find the snapshot process occurring too frequently or causing issues, you can adjust the following: *

automated_cluster_snapshot_interval_seconds

: Default is

600

(10 minutes). Increasing this value reduces the frequency of FE metadata uploads. *

make_snapshot_worker_count

: (BE configuration) Controls the maximum number of threads for snapshot tasks on a BE node (though this is primarily relevant for manual or legacy snapshot tasks). Summary In general, Cluster Snapshot is a low-overhead feature because it leverages the existing shared-data architecture. The performance impact is mostly limited to periodic FE resource spikes for metadata synchronization rather than any sustained impact on query throughput or data loading. *Note: This is currently a Beta feature. It is recommended to monitor FE performance and object storage egress costs when enabling it in a production environment.* References • administration/cluster_snapshot.md • faq/shared_data_faq.md • best_practices/primarykey_table.md

@Rocky does the cluster snapshot backup only the metadata or the data stored inside DBs/tables also

The answer depends on whether you are using the Cluster Snapshot feature (introduced in v3.4.2 for shared-data clusters) or the traditional Backup and Restore functionality. 1. Cluster Snapshot (Shared-Data Clusters) The Cluster Snapshot feature is designed for full disaster recovery of shared-data clusters. It backups both metadata and data. * Metadata: It snapshots the FE state, including catalogs, databases, tables, users, permissions, and loading tasks. * Data: Since data in shared-data clusters is already stored in object storage, the snapshot ensures that the data state at that specific moment is preserved and can be restored. * Workflow: The system automatically maintains a snapshot that follows the cluster state. When you restore, you are restoring the entire cluster to that specific point in time. 2. Backup and Restore (Shared-Nothing Clusters) If you are using the

BACKUP

and

RESTORE

SQL commands (typically for shared-nothing clusters), these also back up both data and the associated metadata. * Data: It creates a physical snapshot of the data (tablets/rows) and uploads it to your backup repository (S3, HDFS, etc.). * Metadata: It saves the schema, partitioning information, and (from v3.4.0 onwards) can also include external catalog metadata, logical views, and UDFs. * Granularity: Unlike a full cluster snapshot, this can be done at the database, table, or partition level. Summary In both cases, StarRocks ensures that you aren't just backing up "labels" or "names" (metadata). The actual row-level data stored inside your tables is included in the backup/snapshot. Key Version Note: * Cluster Snapshot (Full cluster DR) is only available for shared-data clusters starting from v3.4.2. * BACKUP/RESTORE (Granular) is for shared-nothing clusters. Shared-data clusters do not support the standard

BACKUP

and

RESTORE

syntax; they rely on Cluster Snapshots instead. References • administration/management/Backup_and_restore.md

@ahn heesuk @Kevin Cai thoguhts on this one? I'd be open to contributing this back, seems fairly straightforward to implement.

what will do you, stores fe/meta files directly to remote storage?

I've proposed two options: 1. Have cluster snapshot support metadata/fe only, ensure is done on shutdown also, then automate restore in a user friendly manner 2. Store the meta files to remote storage, then cache locally and read from there a. here we need to consider when/why is the metadata written to and how it would affect performance

option 2 will not be as good as expected because either the performance is very bad if leaning to the meta persistency by prioritizing meta writing to remote storage or risk of losing meta if local cache first.

Cool so option 2 is a no go then, question is how trusted/reliable is the graceful shutdown and whether we can implement the backup there, but I guess if the user terminates the pod too early the data will be corrupted anyways, but we could enforce a minimum of some sort The only other option I see is do it from the operator/k8s helm chart side of things and have a prestart/afterstop sidecar or smth like that, before start load the metadata from s3, then after stop store it there

yeah, and even that, it is still not controllable, since the image is not the latest, the FE meta is consistent of IMAGE + WAL, can't do the image creation and upload during the hook of shutdown, it may take long time, unless it is an explicit operation that is known for sure the cluster will be teared down, the availablity is not a concern.

Yep thats what im trying to implement a data lake (s3) with a query/write engine essnetially, and since I dont need everything to be up all the time its most cost efficient to tare down the whole EKS cluster, and if I tare it down attaching and handling persistent PVCs would be a nightmare

if take starrocks just as a query engine, all the data are stored in lake via external catalog, why bother backup starrocks meta, just use a sidecar to init the external catalog and the you go with the query engine.

Ideally I'd like to write the data using starrocks as well, otherwise I'd have to introduce a 2nd software or library into my stack increasing complexity

depends on which external catalog you are using, Starrocks has the ability to write data back to some of the external catalogs, so Starrocks is not required to store any local data at all.

Thanks, will check it out