@Rocky I have 3 Starrocks clusters in version 3.5.17. I'd like to migrate an entire database from 1 to another. My clusters are like so: • destination cluster is shared_data • 1 source cluster is shared_nothing • the other is shared_data as well How should I proceed to migrate the databases ?

Since you are on version 3.5.17, you have several options for migrating databases. The best method depends on whether you are migrating between different storage architectures. 1. Source A (Shared-Nothing) to Destination C (Shared-Data) For this migration path, the recommended approach is the Cross-cluster Data Migration Tool (

starrocks-cluster-sync

). * How it works: This tool automates the process of creating table schemas and synchronizing data from a shared-nothing cluster to a shared-data cluster. * Key Steps: 1. Download the tool:

wget <https://releases.starrocks.io/starrocks/starrocks-cluster-sync.tar.gz>

. 2. Configure

conf/sync.properties

with the FE details of both clusters. 3. Set

include_data_list

to your specific database name. 4. Enable

enable_legacy_compatibility_for_replication = true

on the destination cluster's FE config before running the tool. * Advantage: It handles the schema conversion and data transfer automatically. 2. Source B (Shared-Data) to Destination C (Shared-Data) Because both clusters are shared-data, you have two primary options: Option A: StarRocks External Catalog (Recommended for DB-level) Since you only need to migrate a specific database, using a StarRocks External Catalog is the most straightforward SQL-native method. 1. On Destination C, create a catalog pointing to Source B:

sql
    CREATE EXTERNAL CATALOG source_b
    PROPERTIES (
        "type" = "starrocks",
        "starrocks.fe_http_url" = "http://<source_b_fe_ip>:8030",
        "starrocks.fe_query_port" = "9030",
        "starrocks.user" = "root",
        "starrocks.password" = "your_password"
    );

2. Migrate: Use

CREATE TABLE AS SELECT

or

INSERT INTO ... SELECT

to move the data:

sql
    -- For each table in the database
    CREATE TABLE dest_db.table1 AS SELECT * FROM source_b.source_db.table1;

Option B: Cluster Snapshot (Recommended for Entire Cluster) StarRocks 3.5 introduces Cluster Snapshots specifically for shared-data architecture to replace the traditional Backup/Restore (which is not supported for shared-data tables). * Function: It captures the entire cluster state (metadata and data) and stores it in object storage. * Use Case: This is best if you are migrating everything to a fresh cluster. If Destination C already contains data from Cluster A, do not use this, as restoring a cluster snapshot typically overwrites the target cluster's metadata. Summary Recommendation Migration Path | Recommended Tool A (SN) → C (SD) | Cross-cluster Data Migration Tool (

starrocks-cluster-sync

) B (SD) → C (SD) | StarRocks External Catalog (

INSERT INTO ... SELECT

) Important Notes for 3.5.x: * Backup/Restore Limitation: Standard

BACKUP

and

RESTORE

commands still do not support

CLOUD_NATIVE

(shared-data) tables. Use the External Catalog or Snapshots instead. * Ports: Ensure ports

9030

(Query),

8030

(HTTP), and

8040/9060

(BE/CN) are open between clusters. open between clusters for the migration tool or catalog to work. References • administration/management/Backup_and_restore.md • quick_start/shared-data.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md • faq/Exporting_faq.md • administration/data_migration_tool.md

@Rocky how would I proceed from the shared_nothing to the shared_data cluster as they are in a K8S cluster. I could run the job from a pod instead of my computer? in this case is there a docker image of the migration tool ?

To migrate from a Shared-Nothing to a Shared-Data cluster within a Kubernetes environment, you can indeed run the

starrocks-cluster-sync

tool from a Pod. Is there a Docker image? Currently, there is no official standalone Docker image provided by StarRocks specifically for the

starrocks-cluster-sync

tool. It is distributed as a binary package. However, since the tool is a Java-based utility, you can easily run it using a standard Java or OS image (like

openjdk:11-jre-slim

or

ubuntu

) by downloading the package inside the container. ────────── Recommended Approach: Kubernetes Job The best way to run this in K8S is to use a Kubernetes Job. This allows the migration to run to completion and provides logs you can monitor. 1. Configuration (ConfigMap) First, create a ConfigMap for the

sync.properties

file so you don't have to bake it into an image.

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: migration-config
data:
  sync.properties: |
    # Source Cluster (Shared-Nothing)
    source_fe_host=sr-sn-fe-service.starrocks-sn.svc.cluster.local
    source_fe_query_port=9030
    source_fe_http_port=8030
    source_user=root
    source_password=your_password

    # Destination Cluster (Shared-Data)
    target_fe_host=sr-sd-fe-service.starrocks-sd.svc.cluster.local
    target_fe_query_port=9030
    target_fe_http_port=8030
    target_user=root
    target_password=your_password

    # Database to migrate
    include_db_list=your_db_name

2. The Migration Job You can use a generic image and a

command

to download and run the tool.

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: starrocks-migration
spec:
  template:
    spec:
      containers:
      - name: migration-tool
        image: openjdk:11-jre-slim
        command: ["/bin/sh", "-c"]
        args:
          - |
            apt-get update && apt-get install -y wget
            wget <https://releases.starrocks.io/starrocks/starrocks-cluster-sync.tar.gz>
            tar -xvzf starrocks-cluster-sync.tar.gz
            cp /config/sync.properties starrocks-cluster-sync/conf/
            cd starrocks-cluster-sync
            ./bin/start.sh
        volumeMounts:
        - name: config-volume
          mountPath: /config
      volumes:
      - name: config-volume
        configMap:
          name: migration-config
      restartPolicy: OnFailure

Important K8S Networking Notes * Internal DNS: Use the internal Kubernetes Service DNS names (e.g.,

service-name.namespace.svc.cluster.local

) for the hosts in

sync.properties

. * Pod Access: The Pod running the tool must be able to reach all FE nodes of both clusters and all BE nodes of the source cluster on their respective ports (

9030

,

8030

,

8040

, etc.). * hosts.properties: If your clusters use internal Pod IPs that aren't resolvable across different namespaces or K8S clusters, you may need to provide a

conf/hosts.properties

file to map the IP addresses reported by

SHOW FRONTENDS/BACKENDS

to reachable addresses. Reminder Before running the job, remember to enable legacy compatibility on the destination cluster:

sql
ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication" = "true");

References • administration/data_migration_tool.md • deployment/sr_operator.md • integrations/loading_tools/SMT.md

@Rocky re my first question in this thread, for a shared_data cluster as source, can I migrate the views as well? and can I have the same DB name in my cluster than the one in the external catalog ?

For your migration, you can definitely migrate views, and you can have coexisting databases with the same name thanks to StarRocks' multi-catalog architecture. 1. Migrating Views If you use the Cross-cluster Data Migration Tool (

starrocks-cluster-sync

), views are supported. * Tool Support: The migration tool can automatically synchronize logical views. * Configuration: In your

sync.properties

file, ensure the following parameter is set (it is typically enabled by default in recent versions):

properties
    enable_view_sync=true

* How it works: The tool reads the view metadata from the source and executes the

CREATE VIEW

statement on the destination. * Note: Ensure that the base tables referenced by the views are also being migrated to the same database name on the destination, or the views will fail to execute once migrated. 2. Databases with the Same Name Yes, you can have a database in your local cluster with the exact same name as one in an external catalog. StarRocks uses a three-tier identifier system:

catalog.database.table

. * Local Database: Resides in the

default_catalog

. Its full path is

default_catalog.your_db_name

. * External Database: Resides in your created catalog (e.g.,

source_b

). Its full path is

source_b.your_db_name

. * How to query: You can query both in the same statement without conflict:

sql
    SELECT * FROM default_catalog.my_db.table1 t1
    JOIN source_b.my_db.table1 t2 ON t1.id = t2.id;

* Switching Context: If you run

USE my_db;

, StarRocks defaults to the one in

default_catalog

. To work in the external one, use

USE source_b.my_db;

. 3. Summary of Migration Path (Shared-Data to Shared-Data) Since your Source B and Destination C are both shared_data, you have two solid choices for the views specifically: 1. Using the Migration Tool (Recommended for Automation): Follow the same Kubernetes Job process mentioned earlier. The tool handles both SN → SD and SD → SD migrations identically. It will handle the table data and the view definitions for you. 2. Manual DDL (If only migrating a few views): If you choose to migrate data via

INSERT INTO ... SELECT

using an external catalog, the views won't "auto-copy." You will need to: * Run

SHOW CREATE VIEW source_db.view_name;

on the source. * Copy the output and run it on the destination cluster. Rocky's Tip: For a version 3.5.17 environment, using the

starrocks-cluster-sync

tool is the most robust way to ensure that constraints, views, and data all move together correctly across different cluster architectures. References • administration/data_migration_tool.md • sql-reference/sql-statements/View/CREATE_VIEW.md • data_source/catalog/catalog_overview.md • sql-reference/sql-statements/View/SHOW_CREATE_VIEW.md References • data_source/catalog/query_external_data.md • data_source/feature-support-data-lake-analytics.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • data_source/catalog/unified_catalog.md