@Rocky Is StarRocks' Cross-Cluster Migration Tool based on a physical approach?

Yes, StarRocks' Cross-Cluster Data Migration Tool (also known as

starrocks-cluster-sync

) is based on a physical approach. Instead of using logical SQL commands (like

INSERT INTO SELECT

) to re-process and re-index data, the tool leverages StarRocks' internal replication and cloning mechanism. Here is how the physical approach works: * File-Level Transfer: The tool initiates replication tasks where the target cluster's Backend (BE) nodes communicate directly with the source cluster's BE nodes. They download the actual data files (tablet replicas) and metadata directly. * Bypasses SQL Layer: While the tool uses the SQL layer for metadata operations (like creating database/table schemas), the heavy lifting of data movement happens at the storage engine level by copying physical data files. * Cluster Token Authentication: Because it operates at the storage level, it requires the

token

of the source cluster (found in the FE metadata) to authenticate these low-level data transfer requests. * Efficiency: Because it is a physical migration, it significantly reduces the CPU and memory overhead on the target cluster compared to logical ingestion, as the target cluster does not need to re-encode or sort the data. Key Requirements for this Physical Migration: 1. Network Connectivity: BE nodes in both clusters must be able to communicate with each other over the network (specifically ports like

be_port

and

be_http_port

). 2. Configuration: You must enable

enable_legacy_compatibility_for_replication

on the target cluster to handle version differences. 3. Support: It currently supports migrating data from a shared-nothing cluster to either another shared-nothing cluster or a shared-data cluster. References • administration/data_migration_tool.md • faq/operation_maintenance_faq.md

@Rocky Is it safe even when data is continuously being modified? Does it use a snapshot from the start time?

Yes, it is safe to use while data is being modified, and it does not rely on a single static snapshot taken at the very beginning of the migration. Instead, the tool uses an incremental, version-based synchronization approach. Here is how it handles continuous modifications and consistency: 1. Incremental Sync (Not a single snapshot) The tool does not freeze the source cluster or take a one-time global snapshot. Instead: * Version Tracking: It identifies data versions for each partition (using StarRocks' internal MVCC/versioning mechanism). * Continuous Lag Check: By default, the tool stays alive and regularly checks the source cluster for any new data versions. * Catching Up: If it detects that the target is "lagging" behind the source (i.e., the source has a higher version number for a partition), it initiates a new synchronization task for that specific data. * Final Consistency: If you keep the tool running, it will eventually reach 100% progress once it has copied all versions currently available on the source. 2. Safety During Modifications * Source Cluster: It is perfectly safe to perform

INSERT

,

UPDATE

, or

DELETE

operations on the source cluster during migration. The tool will simply see these as new "lag" and sync the new data in the next cycle. * Target Cluster: You should not load new data into the target tables during the migration process. Doing so can cause version conflicts or data inconsistencies because the tool expects the target state to be driven solely by the source's data versions. 3. How Consistency is Maintained * Transaction-based: The tool opens a transaction for each table sync task. You can even monitor these using

SHOW PROC "/transactions/<db_name>/running";

. * Physical Integrity: Because it copies the physical data files (tablets) of specific versions, the data on the target is a byte-for-byte replica of the source version, ensuring no logical corruption occurs during the transfer. Summary * Approach: Physical and Incremental. * Snapshot Behavior: It uses "point-in-time" snapshots at the task level (syncing specific versions), but the overall process is continuous and catches up to the latest state. * Safe for modifications? Yes, on the source cluster. References • administration/data_migration_tool.md • administration/management/resource_management/Replica.md References • administration/data_migration_tool.md

@Rocky In that case, can the target cluster be kept as a kind of DR cluster on an ongoing basis? In other words, can the migration be left running continuously in production?

While the

starrocks-cluster-sync

tool technically supports continuous synchronization, it is primarily designed for one-time migrations rather than long-term, high-availability Disaster Recovery (DR). If you are looking for a robust, ongoing production DR solution, StarRocks recently introduced Cluster Snapshot (starting from v3.4.2/3.5) specifically for this purpose in shared-data architectures. Comparison for Continuous Usage Feature | ***Cross-cluster Data Migration Tool*** | ***Cluster Snapshot (v3.4.2+)*** Primary Use Case | Moving data from Cluster A to Cluster B (Migration) | Continuous data protection and DR Mechanism | External tool (

starrocks-cluster-sync

) pulling data via BE-to-BE physical transfer | Built-in automated snapshots of metadata and data to object storage Consistency | Table-level eventual consistency | Cluster-level consistent point-in-time snapshots Target State | Target must be "clean" or specific for the tool | Rapid recovery of a full cluster (databases, users, metadata) ────────── Can you leave

starrocks-cluster-sync

running? Yes, but with caveats: * Incremental Mode: The tool does have an incremental mode (it is the default unless

one_time_sync

is set to

true

). It will stay active, poll the source cluster for new data versions, and sync them to the target. * Operational Overhead: Because it is an external process, you must manage its lifecycle (e.g., monitoring the process, ensuring it restarts if it crashes). * Performance Impact: It uses the

replication_threads

and

be_http_port

on BEs. In a long-term production setting, constant data pulling can put steady pressure on the source cluster's IO and network. * Configuration: You must keep

enable_legacy_compatibility_for_replication = true

on the target cluster for as long as the tool is running, which may prevent the target cluster from utilizing some newer version-specific optimizations. The Recommended DR Approach (Shared-Data) If your production environment uses the shared-data architecture, you should use Cluster Snapshot: 1. Automated Snapshots: Enable it via

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON

. 2. Object Storage: It stores the full state (metadata + data) in S3/GCS/HDFS. 3. Cross-Region DR: You can use cloud provider bucket replication to sync the snapshots to a different region and restore a new StarRocks cluster from those snapshots in minutes if the primary site fails. Summary If you just need to keep a second cluster updated for a few days during a migration window, leaving

starrocks-cluster-sync

running is safe and effective. However, for a permanent production DR strategy, the Cluster Snapshot feature is the more modern and integrated choice. References • administration/data_migration_tool.md • administration/cluster_snapshot.md • administration/management/resource_management/Replica.md References • loading/Flink_cdc_load.md • administration/data_migration_tool.md • integrations/loading_tools/SMT.md

@Rocky Cross-cluster migration is also available for shared-data mode — does it actually move the data to a separate object storage, or does it only move the metadata?

When StarRocks' Cross-Cluster Migration Tool (

starrocks-cluster-sync

) is used with a shared-data target cluster, it moves the actual data to the target's separate object storage. It does not just move metadata. Here is how the physical data movement works in shared-data mode: 1. Physical Data Transfer (CN to BE/BE to BE) The migration tool triggers StarRocks' internal replication mechanism. Even in shared-data mode, the target cluster's nodes (CNs/Compute Nodes) perform a physical download. * The Pull: The target nodes communicate directly with the source cluster's BE nodes to download the actual data files (segment files). * The Upload: Once the target node receives these physical files, it uploads them to the target cluster's configured object storage (e.g., S3, GCS, or Azure Blob). 2. Why it moves the data (and not just metadata) * Independence: The goal of the migration tool is to create a fully functional, independent copy of the data. If it only moved metadata pointing to the source's storage, the target cluster would be "tethered" to the source's data lifecycle. If you deleted the source table, the target would break. * Storage Volumes: In the tool's configuration (

sync.properties

), you can specify the

target_cluster_storage_volume

. This tells the target cluster exactly which bucket or storage location should receive the physical files during the migration. * Architecture Requirements: In StarRocks shared-data mode, data is managed by the "Cloud Native" storage engine. For the data to exist in the target cluster, it must be written into the target's storage backend so the target's metadata service (StarOS) can track it. Summary The target cluster becomes a physical replica of the source. After the migration: * The source cluster has its data (on local BE disks or its own object storage). * The target cluster has an identical copy of that data in its own object storage. This physical movement is why the tool is so efficient—it skips the overhead of SQL parsing and re-indexing, but it still results in a completely separate, durable copy of your data on the target side. References • administration/data_migration_tool.md • deployment/shared_data/s3.md • faq/shared_data_faq.md References • administration/http_interface/query_detail.md • using_starrocks/running_queries.md • faq/operation_maintenance_faq.md • administration/management/resource_management/query_queues.md

@Rocky In the case of cross-cluster migration, do files go through the tool during transfer, or do they move directly from BE to BE?

The data files move directly from BE to BE (or from source BE to target CN). They do not go through the migration tool itself. The

starrocks-cluster-sync

tool acts as a coordinator (Control Plane) rather than a data proxy. Here is how the process works: 1. Orchestration: The tool connects to the FEs of both clusters to compare metadata and identify which data versions need to be synced. 2. Instruction: It instructs the target cluster's FE to start a replication task. 3. Direct Transfer: The target cluster's nodes (BEs in shared-nothing or CNs in shared-data) then connect directly to the source cluster's BE nodes via the

be_http_port

(default

8040

) to download the physical segment files. 4. Completion: Once the files are transferred and loaded into the target's storage, the target cluster reports back to its FE, and the tool updates the migration progress. Why this matters: * Performance: Since the tool isn't a bottleneck, the transfer speed is limited only by the network bandwidth and disk I/O between the two clusters' nodes. * Deployment: You can run the tool on a small machine or even a local laptop, as long as it has network access to the FEs. It doesn't need high bandwidth or large amounts of RAM/disk space. * Network Requirements: Your firewall must allow the target cluster's nodes to reach the source cluster's BEs on their HTTP ports (default

8040

). References • administration/data_migration_tool.md References • administration/data_migration_tool.md

@Rocky For cluster upgrades, would a strategy of using cross-cluster migration combined with ProxySQL to gracefully cut over be a viable approach?

Yes, combining cross-cluster migration with ProxySQL for a "Blue-Green" upgrade is a viable and highly recommended strategy for mission-critical environments. This approach minimizes downtime and provides an instant rollback path that a standard rolling upgrade cannot. Here is how to structure this strategy and the key technical considerations: 1. The High-Level Workflow * Provision: Deploy a new StarRocks cluster (the "Green" cluster) at the target version. * Synchronize: Use the

starrocks-cluster-sync

tool in incremental mode. It will perform a full sync followed by continuous incremental updates. * Validation: Direct a small percentage of test traffic (or a shadow read-only load) to the new cluster via ProxySQL to verify performance and query compatibility. * Cutover: Update ProxySQL’s

mysql_servers

table to point the application's hostgroup to the new cluster’s FEs. * Decommission: Once the new cluster is stable, stop the migration tool and decommission the old cluster. 2. Implementation Essentials To make this work in production, pay attention to these specific configurations: * Legacy Compatibility Flag: On the target (new) cluster, you must set:

sql
    ADMIN SET FRONTEND CONFIG ("enable_legacy_compatibility_for_replication" = "true");

This ensures the new version can accept data replicates from the older version's physical format. * ProxySQL Routing: ProxySQL treats StarRocks FEs as MySQL backends. You can swap the clusters by simply updating the

hostname

in the

mysql_servers

table and running

LOAD MYSQL SERVERS TO RUNTIME

. * Incremental Mode: Ensure

one_time_run_mode=false

in the tool’s

sync.properties

. This keeps the migration running continuously, reducing the "data lag" to seconds before you cut over. 3. Critical Gaps (Manual Steps Required) While the migration tool handles data and table schemas, it does not currently synchronize everything. You must manually migrate: * Users & RBAC: User accounts, roles, and resource group permissions are not synced. You should export these from the source (using

SHOW CREATE USER

) and apply them to the target. * Resource Groups: If you use workload management (Resource Groups), these must be recreated on the target. * UDFs: User-defined functions must be manually re-registered on the new cluster. * Materialized View Data: The tool syncs MV definitions (the SQL), but the MVs will need to be refreshed on the target cluster to populate their data. 4. Pros and Cons vs. Rolling Upgrade Feature | Blue-Green (Migration + ProxySQL) | Rolling Upgrade (Standard) Downtime | Near-zero (connection swap only) | Zero (if FE/BE HA is properly configured) Risk | Low. Old cluster remains untouched. | Medium. Irreversible metadata changes occur. Rollback | Instant. Change ProxySQL back to old IPs. | Difficult. Requires metadata/image recovery. Complexity | High (requires 2x hardware + manual metadata sync) | Low (in-place replacement of binaries) Recommendation * Use Rolling Upgrades for minor patch versions (e.g., v3.1.2 to v3.1.5) as they are well-tested and low-risk. * Use the Blue-Green Migration for major architectural shifts (e.g., v2.5 to v3.x) or when moving from shared-nothing to shared-data architectures, where a rolling upgrade is not physically possible. References • administration/data_migration_tool.md • deployment/upgrade.md • administration/management/resource_management/Load_balance.md (ProxySQL details) References • administration/data_migration_tool.md • deployment/upgrade.md