@Rocky Blue and green deployment, use different s3 path. copy blue clusters's s3 data to green clusters's s3 path is ok? How to control data consistency

Hi! I'm Rocky. Dealing with Blue-Green deployments in a shared-data (compute-storage separation) environment requires careful handling of both the physical data on S3 and the metadata stored in the Frontends (FE). Simply copying files from one S3 path to another using S3 tools (like

aws s3 cp

) is not recommended because the FE metadata in the Green cluster will not know those files exist, or the file references in the metadata will still point to the old Blue paths. Here are the recommended ways to handle this while ensuring data consistency: 1. Recommended Method: Backup and Restore This is the safest way to "clone" a cluster to a new S3 path. * How it works: You create a snapshot of your data in the Blue cluster using the

BACKUP

command. Then, you use the

RESTORE

command in the Green cluster. * S3 Path Handling: When you run

RESTORE

in the Green cluster, StarRocks will automatically handle moving/copying the data to the Green cluster's configured storage path and update the new FE metadata to point to the new locations. * Consistency: To ensure 100% consistency, you should set the Blue cluster (or the specific tables) to Read-Only during the backup process. 2. Cross-Cluster Replication (CCR) If you need to keep the Green cluster synchronized with the Blue cluster in real-time before the switchover: * How it works: StarRocks provides a CCR feature (often used for DR) that replicates data from the source (Blue) to the target (Green) cluster. * Consistency: This maintains incremental consistency. Once the Green cluster has caught up, you can stop the replication and point your application traffic to the Green cluster. 3. Shared S3 Path with Cloned Metadata (Alternative) If your Blue-Green deployment is for testing a new StarRocks version and you don't necessarily need a different S3 path: * You can spin up a Green cluster (new FE/BE binaries) and point it to a clone of the Blue cluster's FE metadata. * Warning: Both clusters would be looking at the same S3 data. You must ensure the Blue cluster is stopped or that the Green cluster is treated as read-only to prevent data corruption, as two independent clusters should not write to the same shared-data path simultaneously. Summary of Data Consistency Control To guarantee data consistency during a Blue-Green switch: 1. Stop Ingestion: Stop all load jobs (Routine Load, Stream Load, etc.) on the Blue cluster. 2. Sync Metadata: Perform a final sync (via Backup/Restore or finishing the CCR cycle). 3. Verification: Run checksums or row count queries on both clusters to verify they match. 4. Traffic Switch: Update your Load Balancer or DNS to point to the Green cluster. 5. Resume Ingestion: Start load jobs on the Green cluster. Important Note: Manual S3 file manipulation bypasses the StarRocks metadata layer and will likely lead to "Tablet not found" or "Empty result" errors. Always use the built-in StarRocks commands for data movement.

Can Backup and Restore work well in the shared-data arch ?

Hi there! I'm Rocky. Yes, Backup and Restore works in the shared-data architecture (StarRocks 3.x+), but there are some specific details to keep in mind regarding how it functions compared to the traditional shared-nothing architecture. Here is the breakdown of how it works and its current status: 1. Support Status Since StarRocks v3.1, the

BACKUP

and

RESTORE

commands are supported in the shared-data architecture. You can back up your databases or specific tables to a remote repository (such as S3, OSS, or HDFS). 2. How it works in Shared-Data In a shared-data cluster, your data already resides in object storage. However, the

BACKUP

command still serves several critical purposes: * Point-in-time Recovery: Creating a snapshot of the data and metadata at a specific moment. * Cluster Migration: It is a primary method for moving data between different clusters (e.g., from a shared-nothing cluster to a shared-data cluster, or between two shared-data clusters). * Metadata Consistency: It ensures that the metadata (table schemas, tablet info) and the corresponding data files are captured consistently. 3. Key Considerations * Performance: Because the data is already in remote storage, the "backup" process essentially involves copying those files within the object store (or to another bucket) and exporting the metadata. * Repository Requirement: You still need to create a

REPOSITORY

(using a Broker or direct S3/HDFS integration) to define where the backup should be stored. * Alternative for "Backups": Since the shared-data mode is built on object storage, some users also utilize cloud-native features like Object Versioning or Bucket Snapshots for disaster recovery, although these don't provide the same table-level granularity as StarRocks' native

BACKUP

command. 4. Recommendation If your goal is disaster recovery or data migration, the native

BACKUP

and

RESTORE

work well. If you are looking for frequent snapshots within the same cluster, you might also want to look into the Table Clone feature (if supported for your specific version/use case) or simply rely on the underlying storage's durability. Note: Always ensure that your destination repository has sufficient permissions and that your FE/BE nodes can access it during the process. Is there a specific scenario (like migrating from shared-nothing to shared-data) you are planning? I'd be happy to provide more tailored steps!

@Rocky I just want to deploy blue/green cluster in shared-data arch, Backup & Restore OR DR

Hi! I'm Rocky. For a blue/green deployment or Disaster Recovery (DR) setup in a StarRocks shared-data (Cloud-native) architecture, you have two primary paths depending on your Recovery Time Objective (RTO) and Recovery Point Objective (RPO). Here is a breakdown of how to approach this: 1. Disaster Recovery (DR) via Cross-Cluster Replication (CCR) This is the recommended approach for a "Hot" Blue/Green setup where you want the Green cluster to be a near-real-time replica of the Blue cluster. * How it works: You use the StarRocks Replication Manager tool. It synchronizes metadata and data from the Primary (Blue) cluster to the Standby (Green) cluster. * Shared-Data Benefit: In shared-data mode, the replication process is highly efficient because the data already resides in object storage. * Pros: Low RPO/RTO. You can "promote" the Green cluster to Primary quickly if the Blue cluster fails or during a planned upgrade. * Cons: Requires running two sets of FE/BE nodes (though the Green cluster can be smaller to save costs until failover). 2. Backup & Restore This is a "Cold" or "Warm" approach, better suited for periodic snapshots or migration-style Blue/Green deployments. * How it works: 1. On the Blue cluster, run a

BACKUP

command to an S3/Azure/GDFS repository. 2. On the Green cluster, run a

RESTORE

command from that same repository. * Shared-Data Benefit: In shared-data mode, the

RESTORE

is extremely fast because it primarily restores metadata. The actual data stays in the object storage, and the Green cluster BEs will access it directly. * Pros: Lower cost (no need for a running Green cluster at all times). * Cons: Higher RTO (time to restore metadata) and higher RPO (data is only as fresh as the last backup). 3. The "Compute-Only" Blue/Green (Alternative) Since you are using shared-data architecture, you might not actually need a completely separate cluster for some Blue/Green scenarios (like scaling or isolation). You can use Compute Groups (Warehouses): * You can spin up a "Green" Compute Group while the "Blue" one is running. * Point your application/LB to the Green Compute Group to test performance. * This provides isolation without the complexity of managing two separate sets of FEs (metadata). Recommendation: * For High Availability/DR: Use Cross-Cluster Replication (CCR). It is designed specifically to keep a secondary cluster ready for a Blue/Green switch. * For Upgrades/Testing: Use Backup & Restore if you want a completely isolated environment to test a new StarRocks version before committing. Would you like more details on how to configure the

BACKUP

repository or how to set up the Replication Manager?

@Rocky Can i backup at 3.5.4 cluster, then restore to 4.0.5?

Hello! I'm Rocky. Regarding your question about backup and restore compatibility: In general, StarRocks supports backing up data from a lower version and restoring it to a same or higher version cluster. Therefore, backing up from a 3.x version and restoring to a 4.x version (assuming these versions follow the standard release cycle) is supported by the design of StarRocks metadata and storage compatibility. However, please keep the following points in mind: 1. Forward Compatibility Only: You can move data "forward" (e.g., 3.1 -> 3.3, or 3.x -> 4.x). You cannot restore a backup from a higher version to a lower version cluster. 2. Version Check: Currently, the latest stable releases of StarRocks are in the 3.3.x series. Versions 3.5.4 and 4.0.5 are not yet released in the official StarRocks public roadmap. Please verify the version numbers of your clusters. 3. Testing: Even though it is supported, we strongly recommend performing a test restore with a small subset of data before proceeding with a production migration to ensure that any new features or metadata changes in the higher version do not affect your specific table schemas (especially with complex types or specific index types). General Steps for Backup/Restore: * Create a repository (S3, HDFS, or Azure) on both clusters. * Run

BACKUP SNAPSHOT

on the source cluster. * Run

RESTORE SNAPSHOT

on the target cluster. If you are planning a major version upgrade, you might also consider an in-place upgrade (upgrading FE and BE binaries), which is the standard way to move between StarRocks versions while keeping data intact.

@Rocky SQL Error [5064] [42000]: Table: test_table can not support backup restore, type: CLOUD_NATIVE

The error

SQL Error [5064] [42000]: Table: test_table can not support backup restore, type: CLOUD_NATIVE

occurs because StarRocks shared-data clusters (which use

CLOUD_NATIVE

tables) do not currently support the built-in

BACKUP

and

RESTORE

commands. The

BACKUP

and

RESTORE

functionality is currently designed for shared-nothing (Local Storage) clusters. In a shared-data architecture, data is already persisted in remote object storage (like S3, OSS, or HDFS), but the metadata management and snapshotting mechanism used by the

BACKUP

command are not yet supported for this table type. Recommendations & Workarounds If you need to migrate or "back up" data from a

CLOUD_NATIVE

table, you can consider the following alternatives: 1.

INSERT INTO ... SELECT

: You can create a new table and copy the data using a standard SQL insert.

sql
    CREATE TABLE test_table_backup LIKE test_table;
    INSERT INTO test_table_backup SELECT * FROM test_table;

2. Export to Files (SELECT INTO OUTFILE): You can export the data to external storage (S3/HDFS) in CSV or Parquet format and then re-import it if needed.

sql
    SELECT * FROM test_table
    INTO OUTFILE "<s3://your-bucket/path/data_>"
    FORMAT AS PARQUET
    PROPERTIES (
        "aws.s3.access_key" = "xxx",
        "aws.s3.secret_key" = "xxx",
        "aws.s3.region" = "xxx"
    );

3. Data Life Cycle/Snapshots: Since the data is already in object storage, you may rely on the object storage's own versioning or backup features for the underlying data files, though restoring them to StarRocks would still requires metadata synchronization which is not currently exposed via the

RESTORE

command for shared-data mode. For more details, you can refer to the StarRocks Backup and Restore documentation. References • administration/management/Backup_and_restore.md • sql-reference/sql-statements/backup_restore/BACKUP.md • sql-reference/sql-statements/backup_restore/RESTORE.md

@Rocky So, blue/green deployment can't use backup/restore in the shared-data arch ?

That's correct. Currently, shared-data StarRocks clusters do not support the

BACKUP

and

RESTORE

statements. This is a known limitation in the shared-data (separation of storage and compute) architecture, even in the latest versions (as of v3.4). Because

BACKUP

and

RESTORE

are the primary tools used in shared-nothing clusters to clone data to a new environment, you cannot use this specific mechanism for a blue/green deployment strategy in shared-data mode. How to handle upgrades and isolation in Shared-Data Since you can't use backup/restore for blue/green, here are the recommended alternatives in the shared-data architecture: 1. Rolling Upgrades: StarRocks is designed to support rolling upgrades without downtime. In shared-data mode, you upgrade the Compute Nodes (CNs) first and then the Frontends (FEs). Since compute and storage are decoupled, this process is generally faster and safer than in shared-nothing clusters. 2. Compute Group Isolation (Canary Deployment): Instead of a full blue/green cluster swap, you can use Compute Groups (multi-warehouse). You can spin up a new group of CNs (the "green" group) alongside your existing "blue" group. • You can route a small portion of traffic to the new CN group to test performance or stability. • Once verified, you can shift all traffic to the new CN group and decommission the old one. • Note: This upgrades the compute layer, but they still share the same FE/metadata and storage. 3. Metadata Backup: While you can't "restore" data into a new shared-data cluster via SQL, you can still perform a manual backup of the FE metadata (the

image

and

journal

files). However, this is typically used for disaster recovery rather than a standard blue/green workflow. Summary If your goal for blue/green deployment was data cloning for testing, the lack of `BACKUP`/`RESTORE` makes this difficult in shared-data today. If your goal was zero-downtime upgrades, the rolling upgrade or CN-group isolation strategies are the intended paths for the shared-data architecture. References • quick_start/shared-data.md • quick_start/shared-nothing.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md

Use cluster snapshot if you want to deploy a second cluster based on another cluster on shared data. You can either keep reusing the old bucket or manually copy the data to a new bucket. Snapshot only restore the metadata. https://docs.starrocks.io/docs/administration/cluster_snapshot/