@Rocky Hey team, we are currently running StarRocks v4.0.5 on Kubernetes in Shared-Data mode with an S3 backend. Since the traditional

BACKUP DATABASE ... TO REPOSITORY

command isn't supported in shared-data mode, we have enabled meta-data snapshots but for accidental updates to tables, or corruption in data etc. for taking the full data backup, we come up with: 1. enabled versioning on s3. 2. use aws backup to take backup of s3, which we will be storing for last 2-3days. but since our data is growing as currently we have around 4TB data, this approach is becoming somewhat costly. So, wanted to know from team is there any other recommended way of doing full periodic backups for shared-data clusters?

Hi there! I'm Rocky, and I'd be happy to help you with your backup strategy for StarRocks in Shared-Data mode. In Shared-Data mode (Cloud-Native), the architecture is fundamentally different from Shared-Nothing. Since your data already resides on S3 in an immutable format, you don't necessarily need a traditional "backup" that copies data. Instead, your strategy should focus on Metadata Versioning and Object Storage Lifecycle Management. Here are the recommended ways to handle periodic backups and accidental data loss more cost-effectively than using the AWS Backup service: 1. Leverage "Cluster Snapshots" (Built-in) StarRocks v3.1+ introduced the Cluster Snapshot feature specifically for shared-data clusters. This feature automatically backs up the FE metadata and the current state of data in object storage. * How it works: It ensures that a consistent state of your entire cluster (databases, tables, users, and data pointers) is stored in your S3 bucket. * Recovery: You can restore a new cluster (or the current one) from these snapshots by starting the FE with the

--cluster_snapshot

flag. * Check Status: Query the snapshot status via:

sql
    SELECT * FROM information_schema.cluster_snapshots;

2. "Time Travel" & Metadata Rollback (The "Second-Level" Recovery) For accidental updates or deletes, you don't need a full restore. Shared-data mode supports a high-performance recovery mechanism that rolls back table metadata to a previous healthy version. * Mechanism: StarRocks keeps historical versions of tablet metadata on S3. * Action: You can use the

ADMIN REPAIR TABLE

statement to roll back a table to a specific historical version or timestamp. This is nearly instantaneous because it only changes metadata pointers, not the data files themselves. * Benefit: This avoids the cost and time of moving 4TB of data. 3. Cost Optimization: S3 Lifecycle Policies vs. AWS Backup Since you find AWS Backup costly for 4TB+, you can achieve the same protection at a lower cost using native S3 features: * S3 Object Versioning (Already enabled): This is your primary defense. * S3 Lifecycle Rules: Instead of using the "AWS Backup" service, create Lifecycle Rules on your bucket to: * Transition non-current versions of objects to S3 Standard-IA or S3 Glacier Instant Retrieval after 1–2 days. This significantly reduces storage costs while keeping the data accessible for StarRocks' metadata rollback. * Expire (delete) non-current versions after your required retention period (e.g., 7 days). * Why this is cheaper: You avoid the management fees of the AWS Backup service and leverage cheaper storage tiers for the "deltas" and old versions. 4. Periodic Metadata Backups While you have the cluster snapshot, it's a good practice to periodically back up the FE

meta

directory (or the image files). In Kubernetes: * You can use a

CronJob

to trigger an

ALTER SYSTEM CREATE IMAGE

and then upload the resulting image from the FE's

meta/image

directory to a separate S3 prefix. Summary Recommendation: 1. Keep S3 Versioning enabled but replace the "AWS Backup" service with S3 Lifecycle Policies to move old versions to cheaper tiers (like Glacier Instant Retrieval). 2. Ensure Cluster Snapshots are configured so that your FE metadata is safely stored alongside your data in S3. 3. Use the

ADMIN REPAIR TABLE

command for "Point-in-Time" style recovery for accidental data changes, as it is the most efficient way to recover in Shared-Data mode. Does this help clarify the strategy, or would you like more details on the

ADMIN REPAIR

syntax for your specific version? References • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md

@Rocky here for my shared-data cluster, the `Cluster Snapshots`keeps track of metadata, not actual data so any corruption happens or data gets deleted then it is still not recoverable with this. also As per my understanding `ADMIN Repair`command will only work in-case of data corruption, and not in-case of data deletion. Then tell me do i have any options left which can replicate the behaviour of full cluster + data backup or not, then compared to my current setup?

It sounds like there might be a bit of confusion regarding "Cluster Snapshots" versus the actual Backup and Restore capabilities in StarRocks. If you are looking for a way to recover from actual data deletion or corruption that behaves like a full cluster and data backup, you actually have several robust options beyond

ADMIN REPAIR

. 1. Remote Data Backup & Restore (Full Data + Metadata) Contrary to the idea that snapshots only track metadata, the

BACKUP

command in StarRocks creates a point-in-time snapshot of the actual data files and uploads them to a remote repository (like S3, HDFS, or MinIO). * Behavior: When you run

BACKUP

, StarRocks snapshots the data (SSTables/Segments) and the metadata. This data is physically moved to your external storage. * Recovery from Deletion: If a user accidentally deletes a table or database, you can use the

RESTORE

command to pull that data back from the remote repository. It will recreate the table and restore the data to the exact state it was in at the time of the backup. * Syntax Example:

sql
    BACKUP SNAPSHOT your_db.snapshot_label
    TO your_repository
    ON (your_table);

2. The

RECOVER

Command (For Accidental Deletion) If a database, table, or partition is deleted using the

DROP

command, StarRocks does not physically delete the data immediately. It moves it to a "trash" bin. * How it works: Data is kept in the trash for a configurable period (default is 1 day, controlled by the FE config

catalog_trash_expire_second

). * Usage: You can instantly recover the deleted data without needing to restore from an external backup:

sql
    RECOVER TABLE your_table;

Note: This does not work if you used

DROP TABLE ... FORCE

or

TRUNCATE

. 3. Cross-Cluster Replication (CCR) If you need a "Full Cluster" behavior for Disaster Recovery (DR), you can use Cross-Cluster Replication. * Behavior: This feature synchronizes data from a primary cluster to a secondary cluster at the database or table level. If your primary cluster suffers a catastrophic failure or significant data loss, you can failover to the secondary cluster which contains the replicated data. 4. Shared-Data Architecture Rollback (v3.0+) If you are using the Shared-data (Cloud-native) architecture, StarRocks provides a unique way to handle corruption or accidental loss through Metadata Rollback: * Mechanism: Since data is stored on object storage, StarRocks maintains historical versions of the metadata. You can use an enhanced

ADMIN REPAIR TABLE

to roll back the table's metadata to a "last known healthy" version. * Benefit: This is nearly instantaneous (second-level recovery) even for PB-scale data because it only swaps the metadata pointers back to an older, healthy state on S3. Summary Comparison Feature | Protects Against | Recovery Speed | Notes BACKUP/RESTORE | Data Deletion, Corruption, Site Failure | Slow (Data Movement) | Full physical copy to S3/HDFS. RECOVER |

DROP

statements | Instant | Dependent on

catalog_trash_expire_second

. CCR | Full Cluster Failure | Fast (Switchover) | Requires a second standby cluster. Admin Repair (Shared-Data) | Data Corruption/Inconsistency | Instant | Rollback to historical metadata versions. My recommendation: For a traditional "full backup" safety net, set up a Remote Repository and schedule regular

BACKUP

jobs. This ensures that even if the entire cluster's local disks are wiped, your data is safe on object storage. References • administration/Data_recovery.md • sql-reference/sql-statements/backup_restore/RECOVER.md • faq/shared_data_faq.md • faq/Others.md