@Rocky What are options for StarRocks cluster failover in the context of distinct kubernetes clusters ? I want to maintain two identical live synchronized cluster for quick failover with low downtime.

For achieving a live synchronized StarRocks failover between distinct Kubernetes clusters with low downtime, you have two primary options based on your architecture (Shared-nothing vs. Shared-data). 1. The Cross-cluster Synchronization Tool (Recommended for Live Sync) The most mature method for "live" synchronization is the StarRocks Cross-cluster Data Migration Tool (often referred to as the Sync Tool or CCR). This tool is designed to maintain a synchronized standby cluster. * How it Works: The tool runs as a background process (it can be a separate Deployment in your standby Kubernetes cluster). It connects to the primary cluster's FE and regularly pulls incremental data changes and DDLs to the standby cluster. * Key Configuration: * Set

one_time_run_mode=false

in

sync.properties

to enable continuous incremental synchronization. * Enable compatibility on the target cluster:

ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="true");

. * Pros: Keeps the secondary cluster "warm" and queryable. Since data is already there, failover is nearly instantaneous (just point your application to the new FE LoadBalancer). * Cons: It is a separate tool to manage and requires network connectivity between BE/CN nodes across the two K8s clusters (Ports: 9030, 8030, 9020, 8040, 9060). 2. Cluster Snapshot (Shared-Data Architecture Only) If you are using the Shared-data (compute-storage separated) architecture in StarRocks v3.4.2 or later, you can use the Cluster Snapshot feature. * How it Works: StarRocks automatically persists the entire cluster state (metadata + data) into object storage (S3, HDFS, etc.). In a disaster, you can "spin up" or "point" a second Kubernetes cluster to that existing snapshot in object storage. * Workflow: 1. Enable automated snapshots on the primary:

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME <name>;

. 2. In the second K8s cluster, deploy StarRocks using the Operator and specify the disaster recovery path in the

StarRocksCluster

CRD. * Pros: Very low cost (no extra BE storage cost for the standby cluster until failover). * Cons: Not "live-live" in the sense of a running hot-standby always ready for queries. There is a "restore" step that takes a few minutes to load metadata. 3. Failover Implementation in Kubernetes To manage the actual traffic switch between the two K8s clusters, use the following infrastructure strategies: * Global Load Balancing (GSLB): Use a service like AWS Route53 (with Health Checks), Cloudflare, or a hardware load balancer to point your StarRocks connection string (e.g.,

<http://starrocks.example.com|starrocks.example.com>

) to the active cluster's FE LoadBalancer. * Ingress/Service Mesh: If the clusters are linked (e.g., via Cilium ClusterMesh or Istio), you can use cross-cluster service discovery to route traffic. * Operator Support: The StarRocks Operator includes a

disasterRecovery

field in the Spec. When set, the Operator can help coordinate the "Restoration" mode for a cluster being recovered from a snapshot. Comparison Table Feature | Cross-cluster Sync Tool | Cluster Snapshot (SR 3.4+) Primary Architecture | Shared-nothing / Shared-data | Shared-data only Sync Latency | Seconds (Incremental) | Minutes (Snapshot intervals) Failover Time | Near-instant (Hot standby) | Minutes (Restore metadata) Standby Cost | Full (Running BEs/Storage) | Low (Only object storage) Queryability | Standby is queryable (Read-only) | Standby is not online until restore Rocky's Recommendation: For the lowest possible downtime, use the Cross-cluster Synchronization Tool. Run it as a Pod in your secondary K8s cluster. It will ensure that your secondary BEs have the data locally cached and ready for queries the moment you switch your DNS/Load Balancer. References • administration/data_migration_tool.md