@Rocky In Starrocks 4.1.4 shared-data mode, what are the backup options if cluster-snapshot is not enough? cluster snapshot can be a snapshot of corrupted data, pvc snapshot is not aware of the data in S3 have changed since then and might error

In StarRocks 4.1.4 shared-data mode, while Cluster Snapshot is the primary built-in disaster recovery mechanism, it has the limitation of typically keeping only the "latest" state. If you need to recover from data corruption that occurred before the most recent snapshot, you need options that support historical point-in-time recovery (PITR) or multiple snapshot versions. Here are the primary backup options and strategies beyond a single cluster snapshot: 1. External Backup Tooling (

starrocks-br

) By the 4.x era, the community and partners (like Deep.BI) have introduced specialized CLI tools such as

starrocks-backup-and-restore

. * How it works: This tool manages the orchestration of backups by tracking metadata changes and maintaining a "backup chain." * Key Advantage: It allows you to maintain multiple historical snapshots in your repository (S3/HDFS/GCS) and can resolve backup chains to restore to a specific point in time. This effectively overcomes the "only latest snapshot" limitation of the native automated cluster snapshot. 2. Manual Cluster Snapshot Versioning Since the native

CLUSTER SNAPSHOT

stores both FE metadata and S3 data pointers in a defined object storage path, you can implement a versioning strategy manually or via script: * Mechanism: Before the system overwrites the current snapshot, copy the snapshot directory in your object storage (e.g.,

<s3://bucket/starrocks/snapshots/current/>

to

/snapshots/2026-09-06/

). * Restore: To restore from an older version, you point the new cluster's

cluster_snapshot.yaml

(on the FE) to the specific timestamped sub-path. * Note: This prevents the garbage collector (Vacuum) from deleting data files that are still referenced by your manual "historical" metadata snapshots. 3. Time Travel (Internal Tables) StarRocks 4.1 significantly matured the Time Travel capabilities for internal cloud-native tables. * Usage: You can query historical data using the

AS OF

syntax (e.g.,

SELECT * FROM table AS OF TIMESTAMP '...'

). * Recovery: If data is corrupted by a bad load or accidental delete, you can perform an

INSERT INTO table SELECT ... FROM table AS OF TIMESTAMP

to "rewind" the data to a healthy state without a full cluster restore. * Configuration: Ensure your

storage_volume

has a sufficient

retention_time

configured so that historical data versions aren't vacuumed too early. 4. Export to Open Data Lake Formats (Iceberg/Hive) If the internal table itself is corrupted and you need a "cold" backup: * Strategy: Periodically use the

EXPORT

command or an

INSERT INTO

task to write data to an Iceberg or Hive catalog. * Iceberg Advantage: Iceberg has its own native snapshot and time-travel management. If your StarRocks internal storage is compromised, you can always reload from the Iceberg lakehouse backup. Why avoid PVC Snapshots alone? As you correctly noted, a PVC snapshot of an FE node's

meta

directory only captures the metadata. In shared-data mode, the corresponding data files in S3 are subject to Vacuum (garbage collection). If the cluster continues to run after your PVC snapshot, the vacuum process might delete S3 segment files that the "frozen" FE metadata still thinks exist. Recommendation: If you are in a Kubernetes environment, use the StarRocks Operator's Disaster Recovery mode combined with the

cluster_snapshot

feature, as it is designed to ensure the metadata in the snapshot and the data files in S3 remain synchronized and "pinned" until the snapshot is manually cleared. References • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky starrocks-br supports be, not cn

In StarRocks 4.1.4 shared-data mode, you are correct that the traditional

BACKUP

and

RESTORE

SQL commands (and the

starrocks-br

tool that relies on them) are not supported. These tools were designed for shared-nothing architectures (BE-based) where data is stored locally. If the built-in Cluster Snapshot is insufficient—specifically because it typically captures only the latest state (which could be corrupted)—here are the alternative backup and recovery strategies available: 1. Point-in-Time Recovery (PITR) via Time Travel StarRocks 4.1 shared-data mode supports Time Travel for internal tables. This is often the most efficient way to recover from data corruption without restoring the entire cluster. * Mechanism: You can query the table as it existed at a specific timestamp using the

AS OF

syntax. * Recovery Example:

sql
    -- "Rewind" a table by inserting healthy historical data back into it
    INSERT INTO target_table
    SELECT * FROM target_table AS OF TIMESTAMP '2024-09-05 10:00:00';

* Configuration: You must configure a sufficient

retention_time

(in seconds) on your

storage_volume

or table properties to ensure historical data segments aren't garbage-collected by the vacuum process. 2. Manual Versioning of Cluster Snapshots Since the native Cluster Snapshot feature essentially exports FE metadata and a "manifest" of S3 data files to a specific path, you can create a historical backup chain manually: * Strategy: Write a script to trigger a cluster snapshot, then immediately copy/move the resulting metadata and pointer files in your object storage to a versioned directory (e.g., `s3://bucket/backups/2024-09-06/`). * Prevention of Data Loss: By archiving these pointers, you can restore a cluster to that specific state later. Important: To prevent the actual data segments in S3 from being deleted by the system's "Vacuum" process, you must ensure the retention period for physical data is long enough to cover your backup window. 3. Exporting to Open Table Formats (Iceberg/Hive) For a "cold" backup that is completely independent of the StarRocks metadata engine, you can export data to an external catalog. * Strategy: Use

INSERT INTO ... SELECT

to write data from your internal tables into an Iceberg or Hive table in S3. * Benefit: If the entire StarRocks cluster or its metadata becomes unrecoverable/corrupted, your data remains safely stored in a standard format that can be read by any engine or re-imported into a fresh StarRocks cluster. Iceberg specifically supports its own snapshotting and time-travel. 4. Kubernetes Operator Disaster Recovery If you are running on Kubernetes, the StarRocks Operator provides more robust orchestration for snapshots. It can be configured to automate the creation of these snapshots and can help manage the "Sync" between the FE metadata and the S3 data state, reducing the risk of the "PVC snapshot vs S3 change" error you mentioned. Summary of tool compatibility Feature | Shared-Nothing (BE) | Shared-Data (CN) SQL BACKUP/RESTORE | Supported | Not Supported starrocks-br (CLI) | Supported | Not Supported (uses SQL commands) Cluster Snapshot | Not Applicable | Supported (Primary DR) Time Travel | Limited | Full Support (4.x) Recommendation: For data corruption issues, rely primarily on Time Travel. For total cluster failure, use Cluster Snapshot but implement a manual versioning script to keep historical snapshots in separate S3 prefixes. References • quick_start/shared-data.md • administration/management/Backup_and_restore.md • deployment/shared_data/hdfs.md

@Rocky what happens if the snapshot is corrupted and the FE fails? how do i restore from such state

In the worst-case scenario where the FE fails (local metadata is lost/corrupted) AND your latest Cluster Snapshot is also corrupted (e.g., it captured the corrupted state), you are in a "double-failure" state. Because StarRocks 4.x shared-data mode does not currently support rebuilding metadata from raw S3 segment files (the FE "brain" is required to map table/partition IDs to S3 objects), you must follow these strategies to restore: 1. Restore from a "Versioned" Snapshot (Pre-corruption) The automated Cluster Snapshot only keeps the latest version. To recover from corruption, you must have previously implemented a manual versioning strategy: 1. Preparation (The "Insurance"): Regularly copy the snapshot directory in S3 (e.g., `s3://bucket/starrocks/meta/snapshot/`) to a timestamped folder (e.g., `s3://bucket/backups/2024-09-01/`). 2. The Restore Process: * Deploy a fresh FE node. * Edit

fe/conf/cluster_snapshot.yaml

. Set

cluster_snapshot_path

to the known good URI (e.g., `s3://bucket/backups/2024-09-01/`). * Start the FE with

./fe/bin/start_fe.sh --cluster_snapshot

. * Crucial Requirement: For this to work, the physical data segments in S3 must still exist. You must set a long

retention_time

on your Storage Volume (e.g., 7 days) to ensure the Vacuum process doesn't delete files referenced by your older manual snapshots. 2. S3 Object Versioning / Point-in-Time Restore If you did not manually copy your snapshots, your last line of defense is at the Object Storage layer: * Mechanism: If your S3 bucket has Versioning enabled, you can "rewind" the files in the snapshot directory to a state before the corruption occurred. * Action: Roll back the files in your metadata snapshot prefix to a previous version, then perform the standard Cluster Restore process using the now "clean" snapshot. 3. FE Metadata Recovery Mode If the FE is alive but cannot start due to metadata errors (e.g.,

journal

errors or inconsistent versions), you can force the FE into a recovery state to salvage what is left: 1. Add

metadata_enable_recovery_mode = true

to

fe.conf

. 2. Start the FE. In this mode, the FE will attempt to bypass certain consistency checks and bad journals to bring the cluster online (read-only). 3. Once online, you can check data integrity via

SHOW PROC '/meta_recovery';

. 4. If the data looks salvageable, use

ALTER SYSTEM CREATE IMAGE;

to "checkpoint" a healthy state and restart without recovery mode. 4. Recovery from External Lakehouse Backup If the internal StarRocks metadata and all snapshots are unrecoverable: * Strategy: If you followed the best practice of periodically exporting critical data to Iceberg or Hive (via

INSERT INTO ... SELECT

), you can simply create a fresh StarRocks cluster and re-import or query the data directly from the Lakehouse. This bypasses the corrupted internal StarRocks metadata entirely. Summary of Safety Tiers Protection Level | Mechanism | Can recover from... Cluster Snapshot | Built-in | Total Cluster/FE Hardware Failure. Manual Snapshot Copy | S3 Copy + Long Retention | Data Corruption (Historical Restore). S3 Versioning | AWS/GCS feature | Metadata/File deletion or corruption. Time Travel |

AS OF TIMESTAMP

| Accidental

DELETE

or

UPDATE

(if FE is healthy). Recommendation: For 4.1.4, do not rely solely on the automated cluster snapshot. Implement a daily script to

aws s3 cp

your snapshot prefix to a versioned path and set your Storage Volume

retention_time

to at least 48-72 hours to ensure the physical data stays "pinned" in S3 for those backups. References • deployment/shared_data/s3.md

@Rocky you are saying i can restore from snapshot to a new FE with a new identity? doesn't it break the metadata because the FE is new and not known?

Yes, you can restore a cluster snapshot to a completely new FE with a different identity. StarRocks is designed to handle this through a specific re-mapping process in the

cluster_snapshot.yaml

configuration file. The concern that a "new identity" would break metadata is valid in a normal startup, but the

--cluster_snapshot

restore mode is a special boot sequence that explicitly reconciles the differences between the snapshot's recorded state and your new environment. How it handles the "New Identity" When you restore to a new cluster, you don't just point to the S3 path; you use the

cluster_snapshot.yaml

file to tell the new Leader FE how to "adopt" the old metadata. 1. Re-mapping Nodes: Inside

cluster_snapshot.yaml

, there is a section for node mapping. You can map the IP/port of the FE and CN nodes from the old cluster to the new ones. 2. Storage Volume Re-configuration: You can re-bind the storage volumes recorded in the snapshot to new S3 bucket paths if necessary. 3. Metadata Reset: When you run

./fe/bin/start_fe.sh --cluster_snapshot

, the FE reads the snapshot from S3, clears its local

meta

directory, and initializes itself using the snapshot's metadata image. It effectively "becomes" the cluster described in the snapshot, updated with the new network/storage configurations you provided in the YAML. Steps to Restore to a New FE If your original FE and latest snapshot are corrupted, and you want to use a known good historical snapshot (that you manually copied to a safe S3 path): 1. Deploy a Fresh FE: Do not start it yet. 2. Configure `cluster_snapshot.yaml`: * Set the

cluster_snapshot_path

to your good S3 backup path. * (Optional) If your new FE has a different IP, ensure the configuration reflects where the metadata should now "live." 3. Boot with the Restore Flag:

bash
    ./fe/bin/start_fe.sh --cluster_snapshot --daemon

This flag tells the FE to ignore its own (empty/new) identity and reconstruct its world from the snapshot. 4. Join Other Nodes: * Once the Leader is up, start other FEs as followers (after cleaning their

meta

dirs) using the

--helper

flag pointing to your new Leader. * Start CN nodes after cleaning their local storage directories. Important Limitation: The "Vacuum" Risk Even if you have the metadata snapshot, the data segments in S3 must still exist. * In shared-data mode, the

Vacuum

process periodically deletes S3 files that are no longer referenced by the current metadata. * If your "good" snapshot is 3 days old, but your

storage_volume

has a

retention_time

of only 24 hours, the physical S3 files your snapshot needs may have already been deleted. * Strategy: Always set

retention_time

on your Storage Volumes to be longer than your backup frequency (e.g., if you keep 7 days of snapshots, set retention to 8 days). Summary The restore process is specifically designed to allow "Cross-Cluster Recovery." As long as you provide the mapping in

cluster_snapshot.yaml

, the new FE will successfully take over the metadata from the snapshot without identity conflicts. References • administration/management/Backup_and_restore.md • administration/cluster_snapshot.md • deployment/shared_data/feature-support-shared-data.md • faq/shared_data_faq.md • deployment/shared_data/gcs.md

@Rocky but if i restore from snapshot and the data in S3 has changed since the snapshot was taken, won't it be a problem?

This is a critical point. In a StarRocks shared-data (CN) architecture, the relationship between the Metadata Snapshot and the S3 Data Files is governed by the principle of Immutability. 1. Data Files are Immutable When data "changes" in StarRocks (due to an

UPDATE

,

DELETE

, or background

Compaction

), the existing data files (segments) in S3 are never modified. Instead: * StarRocks writes new segment files for the new data. * The FE metadata is updated to point to these new files. * The old files remain in S3 but are marked as "garbage." 2. The Danger: The "Vacuum" Process The only way a snapshot becomes "broken" is if the Vacuum process deletes the physical S3 files that the snapshot's metadata still points to. If you restore a snapshot from 10:00 AM, but the cluster continued running until 2:00 PM and ran a "Vacuum" at 1:00 PM, that Vacuum might have deleted the files from 10:00 AM because the "live" metadata no longer needed them. If those files are gone, your restored cluster will encounter "File Not Found" errors when querying. 3. How to Prevent This (The "Safety Window") To ensure your snapshots remain valid, you must configure the Vacuum Grace Period to be longer than your intended backup window. * Parameter:

lake_autovacuum_grace_period_minutes

(FE configuration). * Default: Usually 30 minutes (varies by version). * Strategy: If you want to be able to restore a snapshot that is 24 hours old, you must set this parameter to at least

1440

(24 hours). This tells StarRocks: "Even if a file is no longer used by the current metadata, do not delete it from S3 for at least 24 hours." This "pins" the files in S3, ensuring they are still there if you need to roll back to an older metadata state. 4. What happens during Restore? When you restore: 1. Metadata Alignment: The FE loads the metadata from the snapshot. It now "thinks" the world looks exactly like it did when the snapshot was taken. 2. Data Access: When you run a query, the CN looks for the specific S3 file IDs mentioned in that metadata. 3. Result: * Success: If the Vacuum hasn't deleted those files yet, the query succeeds. The "new" data files created between the snapshot and the crash are simply ignored by the restored FE (they will eventually be cleaned up by the next Vacuum cycle of the restored cluster). * Failure: If the Vacuum has deleted those files, the query will fail with an error indicating missing data segments. Summary The data changing is not a problem because StarRocks only adds files; it never overwrites them. The only risk is deletion. Best Practice: * Set

lake_autovacuum_grace_period_minutes

to a value that matches your Disaster Recovery SLA (e.g., 2880 for 48 hours). * Monitor S3 Storage: Increasing this grace period will increase your S3 storage costs, as "garbage" files are kept longer to ensure your snapshots remain functional. References • administration/management/FE_configuration.md • administration/cluster_snapshot.md • faq/shared_data_faq.md • deployment/shared_data/s3.md • administration/management/BE_configuration.md References • faq/shared_data_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/cluster_snapshot.md

@Rocky where can i find the documentation about starrocks Metadata Snapshot Immutability.

You won't find a single document titled "Metadata Snapshot Immutability" because "immutability" is a core architectural principle of the StarRocks Shared-data (Cloud-native) architecture, rather than a separate feature. To understand how snapshots and data safety work together, you should look at these three specific areas of the documentation: 1. Cluster Snapshot Documentation This is the primary guide for the backup and restore mechanism. It explains that the "cluster state" (metadata) is captured and stored in object storage. * Key Concept: A snapshot records the exact IDs of data files (segments) that were valid at that specific moment. * Where to find it: Administration Management > Cluster Snapshot> 2. Auto-Vacuum Configuration (The "Safety Period") The most important part of "immutability" for your snapshots is the Vacuum Grace Period. This is what prevents the physical data from being deleted even if the live cluster moves past it. * Key Parameter:

lake_autovacuum_grace_period_minutes

. * Description: The docs explain this as the "time range for retaining historical data versions." As long as a file is within this grace period, it is immutable and "pinned" in S3, making your older snapshots functional. * Where to find it: FE Configuration Reference 3. Shared-data Architecture Overview The reason data is immutable is that StarRocks uses an LSM-tree-like structure on object storage. When data is modified, it never overwrites an existing S3 object; it always writes a new one. * Key Concept: Historical versions are kept in object storage until the Vacuum process explicitly deletes them. * Where to find it: Shared-data FAQ (See the section "Why does storage usage appear inflated?") Summary for your Safety Plan: If you want to ensure a snapshot from 24 hours ago is always restorable: 1. Read the Cluster Snapshot doc to set up automated or manual metadata backups to S3. 2. Read the FE Config doc and set

lake_autovacuum_grace_period_minutes = 1440

(24 hours). This "freezes" the data segments in S3 for at least one day, ensuring they match your 24-hour-old metadata. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky Cluster snapshot is a complete snapshot of the FE metadata, is there a snapshot per DB option as well? or if we restore from cluster snapshot it means we force all DB's to go back in time

In StarRocks, the answer depends on which architecture you are using. The "Cluster Snapshot" feature and the standard

BACKUP/RESTORE

commands are two different mechanisms with different granularities. 1. Cluster Snapshot (Shared-data / Lake Mode) If you are using the Shared-data architecture (compute-storage separation), the Cluster Snapshot is currently the primary disaster recovery mechanism. * Granularity: Full Cluster only. It is a complete image of the FE metadata (users, privileges, all databases, all tables). * Per-DB Option? No. There is no "per-DB" or "per-table" option for Cluster Snapshots. * The "Back in Time" Reality: Yes, if you restore from a Cluster Snapshot, you are effectively rolling back the entire cluster's metadata to that specific point in time. All databases in that cluster will revert to the state they were in when the snapshot was taken. 2. SQL BACKUP and RESTORE (Shared-nothing Mode) If you are using the Shared-nothing architecture (traditional BE storage), you use the standard SQL commands which offer the granularity you are looking for. * Granularity: Database, Table, or Partition level. * Usage:

sql
    BACKUP DATABASE my_db TO my_repository;
    -- Or for a specific table:
    BACKUP TABLE my_db.my_table TO my_repository;

* Current Limitation: As of StarRocks v3.5, the official documentation states that Shared-data clusters do not yet support the standard SQL

BACKUP

and

RESTORE

commands. They must use the Cluster Snapshot mechanism. Summary Table Feature | Shared-nothing (Local Storage) | Shared-data (S3/Cloud-native) Primary Tool |

BACKUP

/

RESTORE

SQL | Cluster Snapshot (

--cluster_snapshot

) Granularity | Database, Table, Partition | Full Cluster only Restores everything? | No, only what you specify | Yes, rolls back all DBs Workaround for Shared-data DB Recovery If you only want to move one database but are in Shared-data mode, you currently cannot do this via a native snapshot restore. Common workarounds include: 1. Export/Import: Use

INSERT INTO FILES()

to export a specific table to S3 as Parquet and then

INSERT INTO ... SELECT * FROM files()

to bring it back. 2. CTAS from External Catalog: If you have a backup cluster or another environment, you can query it via an External Catalog and use

CREATE TABLE AS SELECT

to pull specific data into your primary cluster without rolling back the whole metadata. References • administration/cluster_snapshot.md • administration/management/Backup_and_restore.md • sql-reference/sql-statements/backup_restore/BACKUP.md • faq/shared_data_faq.md References • administration/cluster_snapshot.md • faq/Exporting_faq.md • administration/management/Backup_and_restore.md