Hi Everyone, @Rocky We are looking for guidance on a disaster recovery approach, specifically for a shared-data deployment across two regions. We are weighing two options and would value some recommendations on both. Option 1: a single cluster spanning both regions Deploying FEs and CNs across both regions, so that the loss of one region leaves a surviving set of FEs and CNs still able to serve. Setting aside the latency concerns for a moment, our questions are: 1. Is this a supported topology at all, and does anyone run it? 2. When a region goes down, how does StarRocks actually behave with only a partial set of FEs and CNs remaining? Specifically: - What happens to the FE quorum if the surviving region held the minority of followers? Is there a supported recovery path, and is it automatic or manual? - Do the surviving CNs continue serving queries during the FE outage, or does everything stop? - Is there a recommended FE distribution across two regions, or is this simply not advised? Option 2: two independent clusters One cluster per region, with the second cluster brought up on either the same bucket or a replicated copy of it in the second region. Our main question here is about state: 3. Is there a supported way to back up the FE metadata and restore it into a second cluster in another region, such that the cluster resumes correctly against the existing bucket or a replica of it? 4. Is there any mechanism that captures the FE metadata together with the object storage state as a single consistent point, or are these necessarily two separate procedures that have to be coordinated manually? 5. If the object store is replicated asynchronously to the second region, the metadata may reference objects that have not yet arrived. Is there a supported way to establish a consistent recovery point across the metadata and the bucket, or a tool to detect and reconcile missing objects after a restore? 6. In the storage volume definition, are the endpoint and bucket properties mutable after the volume already holds data? We would like to know whether a cluster can be repointed at a replica bucket at a different endpoint, or whether that has to be arranged up front via a stable DNS name. Finally, if there are users running shared-data mode in production with a DR requirement, it would be very helpful to learn how they are handling this in practice. Thanks!

For a shared-data (cloud-native) deployment across two regions, here is the guidance based on StarRocks' architecture and documentation. Option 1: Single Cluster Spanning Two Regions (Stretch Cluster) 1. Is this a supported topology? Yes, it is technically supported, but it is rarely recommended for cross-region setups due to two main constraints: * Consensus Latency: StarRocks uses BDBJE (Berkeley DB Java Edition) for metadata consensus. Every metadata change (DDL, data loading, etc.) requires a round-trip acknowledgment (ACK) from a majority of Follower FEs. High cross-region latency will directly impact the performance of all write operations. * Clock Synchronization: StarRocks requires FE clocks to be synchronized within a 5-second delta (

max_bdbje_clock_delta_ms

). 2. Behavior during a regional outage: * FE Quorum: StarRocks requires more than half of the Follower FEs to be alive to elect a leader. In a two-region setup (e.g., 2 Followers in Region A, 1 in Region B), if Region A goes down, the surviving region (Region B) will have only a minority and cannot automatically elect a leader. * Recovery Path: Recovery is manual. You must set

bdbje_reset_election_group = true

in the surviving FE's

fe.conf

to force it to become the leader and reset the consensus group. * CN Behavior: CNs cannot serve queries during an FE outage. CNs rely on the FE Leader for query planning and metadata. If there is no FE Leader, queries will fail. * Recommended Distribution: Because a two-region split cannot provide automatic failover for a majority-based consensus, it is simply not advised for automatic DR. A 3-region deployment (1 Follower per region) is the standard for automatic regional HA. ────────── Option 2: Two Independent Clusters 3. FE Metadata Backup and Restore: Currently, the SQL-based

BACKUP

and

RESTORE

commands do not support shared-data clusters. For DR, you must perform a manual file-level backup of the FE

meta_dir

. * Restore Path: You can restore by copying the

meta_dir

to a new FE in the second region and starting it with

bdbje_reset_election_group = true

. 4. Consistency Mechanism: There is no built-in mechanism that captures FE metadata and object storage state as a single atomic snapshot. These are separate procedures. 5. Reconciliation (Handling Asynchronous Replication): If the object store is replicated asynchronously, the restored FE metadata may reference data files that haven't arrived in the secondary region. * Reconciliation Tool: StarRocks provides the

ADMIN REPAIR TABLE

statement specifically for shared-data tables. It allows the system to scan historical metadata versions and roll back the Tablet metadata to the most recent "healthy" version where all referenced files actually exist in the bucket. This is an ultra-fast, metadata-only operation. 6. Storage Volume Mutability: The

TYPE

,

LOCATIONS

, and path-related properties of an existing Storage Volume cannot be altered (they are immutable). Only credential-related properties (like access keys) can be modified via

ALTER STORAGE VOLUME

. * Recommendation: To point a cluster at a replica bucket at a different endpoint, you must either: 1. Use a stable DNS name (e.g.,

dr-bucket.internal

) that you can repoint at the network level. 2. Create a new Storage Volume in the second cluster that points to the new bucket, though this usually requires re-creating the table schemas in the second cluster to use that volume. Summary Recommendation In practice, most users with strict DR requirements for shared-data mode use Option 2 with a "Cold Standby" or "Warm Standby" approach: 1. Replicate the Object Store at the storage layer (e.g., S3 Cross-Region Replication). 2. Periodically backup the FE

meta_dir

to the second region.