@Rocky Can two starrocks clusters point to the same storage volume?

Yes, two StarRocks clusters (or multiple compute groups) can point to the same storage volume, but the answer depends on how you define "cluster" and what you want to achieve: *1. Multi-Warehouse (Compute Clusters) - Supported* In StarRocks 3.x shared-data (compute-storage separation) mode, you can have one logical StarRocks instance (one set of FEs) manage multiple compute clusters (groups of CN nodes). * How it works: All compute clusters share the same metadata and the same storage volume (S3, HDFS, etc.). * Use case: This is the recommended way to achieve resource isolation. For example, you can have one compute cluster for data loading and another for user queries, both pointing to the same data volume. *2. Independent Clusters (Separate FEs) - Not Recommended for the same path* If you have two entirely independent StarRocks clusters (each with its own set of FE nodes and its own metadata), you should not point them to the exact same storage path/prefix for their internal data. * Why: Each FE cluster maintains its own independent metadata (transaction IDs, table schemas, file mappings). If two independent clusters write to the same path, they will not see each other's updates, and they may overwrite or delete each other's data files during background tasks like compaction or garbage collection, leading to data corruption. * Solution: Give each independent cluster its own unique sub-path (prefix) within the same bucket (e.g.,

<s3://my-bucket/cluster1/>

and `s3://my-bucket/cluster2/`). *3. Sharing Data via External Catalogs - Supported* If you want two independent clusters to access the same data files simultaneously, you can use External Catalogs. * How it works: Both clusters can point to the same external storage (like an Iceberg, Hive, or Delta Lake table on S3). * Use case: Since the metadata is managed externally (e.g., via AWS Glue or Hive Metastore), both StarRocks clusters can safely read from and even write to the same data volume without conflict. Summary Table Scenario | Supported? | Description Compute Clusters | Yes | Multiple groups of CN nodes under one FE cluster sharing the same volume. Independent FEs (Same Path) | No | Leads to metadata conflict and data corruption. Independent FEs (Unique Paths) | Yes | They can share the same bucket but must use different prefixes. External Catalogs | Yes | Both clusters can access the same Hive/Iceberg/Delta data on the same volume. If you are looking to scale compute independently or isolate workloads, the Multi-warehouse (Compute Cluster) feature in StarRocks 3.x is the best approach. References • introduction/Architecture.md • deployment/shared_data/s3.md • quick_start/shared-data.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/hdfs.md

@Rocky I am using shared data mode with an s3 storage volume so is that a bad idea to point two clusters to it to read the same data?

In short: Yes, pointing two independent StarRocks clusters (separate FE clusters) to the exact same S3 storage path for shared-data mode is a bad idea and can lead to data corruption. However, if your goal is to have multiple sets of compute resources accessing the same data, StarRocks has a specific feature for this called Multi-warehouse (Compute Groups). Here is a breakdown of why independent clusters shouldn't share the same path and how you should actually set this up: 1. Why independent clusters shouldn't share the same S3 path In StarRocks' shared-data architecture (Cloud-native tables), the Frontend (FE) manages the metadata, such as which S3 files (segments) belong to which table and version. * Metadata Inconsistency: If Cluster A writes data to S3, Cluster B’s FE has no way of knowing those files exist because its own metadata hasn't been updated. * Data Corruption: StarRocks clusters perform background tasks like Compaction and Garbage Collection (GC). If Cluster A decides a file is no longer needed and deletes it from S3, but Cluster B was still using it, Cluster B will fail. If both try to compact the same data, they will overwrite each other's work. 2. The Recommended Solution: Multi-warehouse (Compute Groups) If you want two "clusters" (sets of compute nodes) to read the same data, you should use a single FE cluster to manage multiple Compute Groups (also known as Multi-warehouses). * How it works: You have one set of FEs (the "brain") and multiple groups of Compute Nodes (CNs). * Benefits: * Resource Isolation: You can have one CN group for high-priority BI queries and another for heavy data loading. * Shared Metadata: Since they share the same FE, there are no metadata conflicts or data corruption risks. * Elasticity: You can scale each compute group independently. 3. When it IS okay to point multiple clusters to S3 It is perfectly safe to point multiple independent StarRocks clusters to the same S3 data if: * External Catalogs: You are using an External Catalog (like Iceberg, Hive, or Hudi) to read open-format data (Parquet/ORC). In this case, the metadata is managed by an external service (like AWS Glue or Hive Metastore), and StarRocks treats the S3 bucket as a read-only or externally managed data lake. * Read-Only "Storage Volumes": If you are only using S3 as a source for

INSERT INTO ... SELECT * FROM FILES(...)

(Table Value Functions), multiple clusters can read those source files simultaneously without issue. Summary * Don't point two separate FE clusters to the same

aws_s3_path

for their internal shared-data storage. * Do use Compute Groups within a single cluster if you need multiple compute pools for the same data. * Do use External Catalogs if you want separate clusters to access a shared Data Lake. References • deployment/shared_data/s3.md

@Rocky Okay is it possible to clone an s3 storage volume and use it in another independent cluster then? Or what if I am using AWS and I just copy the current storage volume bucket from one bucket to another? Thoughts?

In StarRocks (specifically in Shared-Data mode), the short answer is no, simply copying the S3 bucket and pointing a new cluster at it will not work. Here is the breakdown of why this is the case and how you can actually achieve your goal. 1. The Metadata Dependency In StarRocks, data storage is split into two parts: * Data Files: These are the actual segments and parity files stored in your S3 bucket. * Metadata: This is the critical "map" that tells StarRocks which files in S3 belong to which table, partition, or tablet. This metadata resides in the FE (Front-End) nodes (in their memory and local

meta/

directory). If you copy an S3 bucket to a new location and start a brand-new independent cluster pointing to that new bucket, the new cluster’s FE will have an empty metadata catalog. It will have no idea that those files in S3 represent existing tables. It won't "auto-discover" them. 2. What happens if you try? If you point a new cluster at a bucket containing data from an old cluster: * The new cluster will start up fine but will show no databases or tables. * If you try to create a table with the same name, it might create new files in that bucket, potentially leading to a mess or naming collisions, but it still won't see the old data. 3. How to actually "Clone" or "Migrate" If you want to use data from one cluster in another independent cluster, you have three main options: A. Backup and Restore (Recommended) This is the official way to move data between clusters. * How it works: You run a

BACKUP

command in the source cluster. This creates a snapshot that includes both the metadata and the data. You then run a

RESTORE

command in the target cluster. * Version Note: For shared-data (Cloud-Native) tables, full support for Backup/Restore was finalized in version v3.4.0. If you are on an older version, this might only be available for shared-nothing tables. B. Data Export/Import (Standard) If you only need a subset of data, you can export the data to S3 (using

EXPORT

or

INSERT INTO FILES

) in a format like Parquet, and then use

LOAD

(Stream Load, Broker Load) or

INSERT INTO SELECT

from an External Table in the new cluster. C. External Catalog (Data Lake approach) If your goal is to have multiple independent clusters querying the same data, you might consider storing the data in an open format (like Apache Iceberg or Hive/Hudi) on S3. * Both clusters can then create an External Catalog pointing to the same S3 location. * Since the metadata for Iceberg/Hive is stored in a shared Glue or Hive Metastore, both StarRocks clusters will see the same tables instantly. Summary of your "Thoughts" * Cloning an S3 Volume: Creating a new

STORAGE VOLUME

in a new cluster pointing to an old bucket's path won't bring the tables over; it just gives the new cluster permission to write to that path. * AWS S3 Copy: Copying the bucket is a great way to duplicate the physical data for safety, but without a metadata migration (via

BACKUP/RESTORE

), the StarRocks FE won't be able to use it. Rocky's Tip: If you are trying to scale compute or isolate workloads (e.g., a "Prod" cluster and a "Dev" cluster), look into Compute Groups if you are on a managed platform, as this allows multiple compute clusters to share the same storage and metadata without copying anything! References • deployment/shared_data/s3.md • administration/data_migration_tool.md • deployment/shared_data/minio.md • quick_start/shared-data.md

@Rocky The documentation for backup and restore says shared data does not support backup and restore

You are correct. In StarRocks, the traditional

BACKUP

and

RESTORE

SQL commands are currently not supported for shared-data clusters. These commands were originally designed for the shared-nothing architecture to move data between local disks and remote repositories. However, since shared-data clusters already store their data files in remote object storage (like S3, GCS, or HDFS), the approach to backup and disaster recovery is different: 1. New Feature: Cluster Snapshot (v3.4.2+) Starting from StarRocks v3.4.2, a new feature called Cluster Snapshot was introduced specifically for shared-data clusters. * What it does: It captures the entire state of the cluster, including both the data (already in object storage) and the metadata (FE state like catalogs, databases, users, and permissions). * Why it's needed: In shared-data mode, while your data is safe in S3/HDFS, the FE metadata is still managed locally. Cluster Snapshot ensures that a full recovery is possible even if all FE nodes are lost. * Status: It is currently a Beta feature and is the recommended way to handle disaster recovery for shared-data architectures. 2. Inherent Data Durability Because shared-data clusters use cloud-native storage (S3, Azure Blob, etc.), your data files already benefit from the high durability and availability provided by the cloud provider (e.g., 99.999999999% durability). Many users rely on the cloud provider's own versioning or cross-region replication features at the bucket level for additional protection. Summary of Alternatives Feature | Shared-Nothing (Classic) | Shared-Data (Cloud-Native) BACKUP / RESTORE | Supported | Not Supported Cluster Snapshot | Not Supported | Supported (from v3.4.2) Data Export |

EXPORT

or

INSERT INTO SELECT

|

EXPORT

or

INSERT INTO SELECT

If you are looking to migrate data or create a point-in-time copy, you can still use the

EXPORT

command or

INSERT INTO FILES()

to write data out to a separate location in Parquet or ORC format. References • administration/cluster_snapshot.md • deployment/shared_data/gcs.md • deployment/shared_data/feature-support-shared-data.md • faq/shared_data_faq.md • deployment/shared_data/azure.md

@Rocky Can I restore another cluster from a snapshot another made?

Yes, you can restore data to a different StarRocks cluster using a snapshot created by another cluster. There are two primary ways to achieve this depending on whether you want to restore specific data (databases/tables) or the entire cluster state. 1. Standard Backup and Restore (Tables/Databases) This is the most common method for migrating data between clusters or restoring specific datasets. * How it works: You create a snapshot in the source cluster using the

BACKUP

command and then use

RESTORE

in the target cluster. * Key Requirement: The target cluster must have a repository created with the exact same name and location (endpoint and bucket/path) as the one in the source cluster. * Steps: 1. In the source cluster, create a snapshot:

BACKUP SNAPSHOT ...

2. In the target cluster, create a repository pointing to the same storage location. 3. In the target cluster, run the restore command:

sql
        RESTORE SNAPSHOT your_backup_name
        FROM your_repo_name
        DATABASE your_db
        PROPERTIES("backup_timestamp" = "2024-xx-xx-xx-xx-xx");

* Note: You do not need to create the table/database in the target cluster beforehand; the restore process will create them automatically. 2. Cluster Snapshot (Full Cluster Recovery) For shared-data clusters, StarRocks supports a "Cluster Snapshot" feature that allows you to restore the entire state of a cluster (including metadata and data) into a new cluster. * Use Case: Full disaster recovery or cloning an entire shared-data cluster. * Key Requirement: You must modify the

fe/conf/cluster_snapshot.yaml

file on the Leader FE node of the new cluster before starting it. * Steps: 1. Prepare the

cluster_snapshot.yaml

with the URI of the snapshot and the new cluster's node information (FE/CN hosts). 2. Start the Leader FE with the

--cluster_snapshot

flag:

bash
        ./fe/bin/start_fe.sh --cluster_snapshot --daemon

3. Start other nodes after cleaning their local metadata/storage directories. Important Considerations * Version Compatibility: Ensure the target cluster version is compatible with the source. For example, Primary Key tables cannot be restored to versions earlier than v2.5. * Privileges: Backup and Restore operations do not automatically migrate user accounts, privileges, or resource groups. You will need to reconfigure these in the new cluster. * Shared Storage: Both clusters must have access to the same remote storage (S3, HDFS, Azure Blob Storage, etc.) where the snapshots are stored. References • administration/cluster_snapshot.md • administration/management/Backup_and_restore.md • sql-reference/sql-statements/backup_restore/RESTORE.md

@Rocky If one starrocks cluster generate the starrocks cluster snapshot in its external volume can I copy that file to restore the cluster in another storage volume or something?

Yes, you can copy a cluster snapshot to another storage location and use it to restore the cluster (cross-cluster recovery). This feature is primarily designed for shared-data clusters. Here is how you can achieve this and the requirements involved: 1. Copy the Snapshot Files To restore data in a new cluster or a different storage volume, you must copy all files under the original storage path (including the metadata and data objects) to the new destination path in your object storage (S3, HDFS, etc.). 2. Configure

cluster_snapshot.yaml

In the Leader FE node of the new cluster, you must modify (or create) the

fe/conf/cluster_snapshot.yaml

file. This file tells StarRocks where to find the snapshot and how to map the storage. Key sections in the YAML: *

cluster_snapshot_path

: Point this to the URI of the snapshot you copied (e.g., `s3://new-bucket/path/to/snapshot`). *

storage_volume_name

: This must be identical to the name of the storage volume in the original cluster. *

storage_volumes

: Define the storage volume properties (type, location, credentials) for the new location. Even if the path is different, the

name

field here must match the original cluster's volume name. 3. Restore Process 1. Stop the new cluster (or ensure it's a clean environment). 2. Start the Leader FE with the snapshot flag:

bash
    ./fe/bin/start_fe.sh --cluster_snapshot --daemon

3. Start other FE/CN nodes: Clean their

meta

and

storage_root_path

directories first, then start them as helpers to the leader. Important Notes: * Storage Volume Name constraint: As mentioned, the logical name of the storage volume must match the original. If your original volume was named

main_storage

, your new cluster's YAML must also name it

main_storage

, even if it points to a different physical bucket. * Architecture: This specific "Cluster Snapshot" method is for shared-data clusters. If you are using a shared-nothing architecture, you should use the standard

BACKUP

and

RESTORE

commands via a

REPOSITORY

. * One Snapshot: StarRocks typically maintains only the latest automated snapshot. Ensure you copy the snapshot you intend to keep before it is overwritten by a newer one. For more detailed templates of the YAML file, you can check the

administration/cluster_snapshot.md

section in the official documentation. References • administration/cluster_snapshot.md • faq/Exporting_faq.md • administration/management/Backup_and_restore.md

@Rocky I thought "BACKUP SNAPSHOT" did not work on shared data architectures/

You are correct. The traditional

BACKUP SNAPSHOT

and

RESTORE SNAPSHOT

commands are not supported for StarRocks shared-data (compute-storage separation) clusters. Here is how disaster recovery and data protection work in the shared-data architecture: 1. The "Cluster Snapshot" Feature In a shared-data cluster, because the data files are already stored in object storage (like S3 or GCS), the "backup" process is handled differently. StarRocks provides a Cluster Snapshot mechanism specifically for this architecture: * Purpose: To ensure the full cluster state—including both data (in object storage) and FE metadata—is protected. * How it works: The system automatically maintains a snapshot of the cluster's state. It periodically uploads the FE metadata to the object storage. * Recovery: If the cluster fails, you can restore a new cluster directly from the snapshot stored in object storage. 2. Why

BACKUP SNAPSHOT

is not used for Shared-Data The

BACKUP SNAPSHOT

command was designed for shared-nothing architectures to physically copy data from local BE disks to a remote repository. In shared-data mode: * Data is already remote. * Copying petabytes of data from one object storage location to another using the old

BACKUP

logic would be redundant and inefficient. 3. Alternative: Data Export If you need to move specific data between clusters or create an archival copy of a specific table in a shared-data environment, you can use: * EXPORT: To export table data to files (CSV, Parquet) in object storage. * INSERT INTO ... SELECT: Between different catalogs or clusters. Summary: Your intuition was right. If you are running a shared-data cluster and need disaster recovery, you should look into the Cluster Snapshot configuration in your FE properties rather than the

BACKUP SNAPSHOT

SQL command. References • administration/cluster_snapshot.md • introduction/Architecture.md • sql-reference/sql-statements/backup_restore/BACKUP.md • faq/Exporting_faq.md • administration/management/Backup_and_restore.md

@Rocky I am working on disaster recovery and need to know if I can move a shared data storage volume to another if I needed to move regions or entire services basically to something like google cloud

In StarRocks (specifically in Shared-data mode, also known as Compute-Storage separation), moving a shared data storage volume to another region or a different service (like Google Cloud GCS) is possible, but there are important architectural details and limitations you should be aware of for disaster recovery (DR). 1. Object Storage Compatibility StarRocks Shared-data mode supports S3-compatible storage. Since Google Cloud Storage (GCS) provides an S3-compatible XML API, you can move your data from another provider (like AWS S3) to GCS. * Authentication: In GCS, you would use HMAC keys (Access Key / Secret Key) to connect. * Configuration: You would set the

aws_s3_endpoint

to

<https://storage.googleapis.com>

. 2. The "Storage Volume" Limitation Starting from StarRocks v3.1, storage is managed via Storage Volumes. If you are planning to "move" an existing volume by pointing it to a new location (different bucket or region): * Location is Immutable: According to the StarRocks documentation, once a Storage Volume is created, its

LOCATIONS

and

TYPE

cannot be altered. * Read-Only Risk: If you attempt to change the underlying storage path configuration for an existing cluster, the existing databases and tables associated with that volume may become read-only, and you will not be able to load new data into them. This is because the FE (Frontend) metadata stores specific paths to the data files. 3. Disaster Recovery Strategies For a true disaster recovery or migration scenario where you move regions or clouds: Option A: Metadata + Object Storage Replication (Recommended for DR) The most robust way to move an entire "service" is to ensure both the FE Metadata and the Object Storage Data are synchronized to the target region. 1. Sync Object Storage: Use tools like

gsutil

(for GCS) or

rclone

to replicate your bucket from the source region to the destination region. 2. Sync FE Metadata: The "source of truth" for where data lives is in the FE's BDBJE (metadata). For a total region failover, you would need a backup of the FE's

meta

directory. 3. Point to New Storage: If the bucket name and path remain identical and you are simply changing the endpoint/region in the

fe.conf

, the cluster can theoretically start up in the new environment. However, if the bucket name changes, the metadata will still point to the old bucket name. Option B:

INSERT INTO ... SELECT

(Migration) If you are performing a planned migration to Google Cloud: 1. Create a new Storage Volume in the new region/service. 2. Create new tables using that volume. 3. Load data from the old tables to the new ones using

INSERT INTO new_table SELECT * FROM old_table

. This ensures the metadata is correctly mapped to the new GCS location. Option C:

ADMIN REPAIR TABLE

(File-level DR) If you lose files or have consistency issues during a move, StarRocks (v3.2.4+) provides a

ADMIN REPAIR TABLE

mechanism. This is a "second-level recovery" that rolls back the table metadata to a previous healthy version where all files were present in the object storage. This is useful if a migration goes wrong and some files are missing. Summary * Can you move the volume? Yes, by copying the data at the object storage level. * Can you simply update the existing StarRocks cluster to point to the new location? Only for credentials. Path-related changes (

LOCATIONS

) for an existing volume are not supported and will likely result in a read-only state. * Best Practice: For moving regions/services, it is often safer to set up a new cluster in the destination, define the new GCS storage volume, and migrate the data via

INSERT INTO SELECT

or by restoring from a full FE metadata snapshot if the bucket structure is identical.

@Rocky If I want to implement backup strategy for starrocks 3.3.12 running in shared data mode, what would be ideal since I cannot use backup and restore and cannot use cluster_snapshot feature. If I simply enable versioning on s3 bucket where starrocks data is residing, would that be enough to restore an old table or database? how would that happen

Hi, I'm Rocky! I've researched the backup options for StarRocks 3.3.12 in shared-data mode. You are correct that in version 3.3.12, the standard

BACKUP

and

RESTORE

commands are not supported for shared-data clusters, and the Cluster Snapshot feature is only available starting from version 3.4.2. Regarding your proposed strategy: Is S3 Versioning enough? No, enabling S3 versioning alone is not sufficient to restore a table or database. In StarRocks shared-data mode, your data is stored in S3, but the metadata (which files belong to which table, tablet mappings, and data versions) is stored in the Frontend (FE) nodes (and the internal StarOS service). If you were to roll back an S3 bucket to a previous version using S3 versioning: 1. Metadata Mismatch: The FE metadata would still point to the "future" versions of the data files. It would expect certain file IDs and versions that no longer exist or have been overwritten in S3. 2. Cluster Inconsistency: This will likely lead to "file not found" errors or data corruption during queries, as the FE's view of the storage will not match the physical files in S3. Recommended Backup Strategies for v3.3.12 Since you are on 3.3.12, here are the most effective alternatives: 1. Logical Backups (Recommended) The most reliable way to back up specific tables or databases is to perform a logical export of the data: * `INSERT INTO ... SELECT`: You can create an external table (e.g., using an Iceberg catalog or a Hive catalog) and periodically run

INSERT INTO external_table SELECT * FROM internal_table

. *

EXPORT

command: Use the

EXPORT

statement to dump table data into S3 in Parquet or CSV format. This allows you to reload the data using

INSERT INTO ... SELECT files()

if needed. 2. Using External Tables for Critical Data If point-in-time recovery is a hard requirement, consider storing that data in an External Catalog (like Apache Iceberg) instead of a Cloud Native table. StarRocks can query Iceberg tables with high performance, and Iceberg has its own native snapshot management and Time Travel capabilities that are independent of the StarRocks FE metadata. 3. Accidental Deletion Protection (Recycle Bin) StarRocks has a built-in "Recycle Bin" for shared-data mode. If a table or database is dropped by mistake: * The data is not immediately deleted from S3. * You can recover it using the

RECOVER TABLE

or

RECOVER DATABASE

command within a certain time window (controlled by the FE configuration

catalog_trash_expire_second

). 4. Metadata + Storage Snapshot (Full Cluster Recovery Only) If you need a "disaster recovery" plan for the entire cluster: 1. Take a snapshot of the FE metadata (by copying the

meta/

directory or triggering a checkpoint). 2. Simultaneously take a snapshot/version-point of the S3 bucket. _ _Note: This is a complex manual process and is only intended for restoring the whole cluster to a specific point in time, not individual tables.* Summary The ideal strategy for your version is to use logical backups (Exporting to S3 or syncing to an external Iceberg table). If you require automated, built-in cluster-level snapshots, I recommend planning an upgrade to v3.4.2 or higher or later. References • quick_start/shared-data.md • deployment/shared_data/s3.md • administration/management/Backup_and_restore.md • deployment/shared_data/feature-support-shared-data.md