Hi @Rocky, could you recommend some replication tool that is commonly used for starrocks? We try airbyte, but it loads the data in starrocks as a raw json file, and need additional step to parse into table.

For your requirement of replicating data into structured tables (rather than raw JSON), there are several well-supported tools in the StarRocks ecosystem that handle schema mapping and structured data loading directly. Here are the most commonly used replication tools for StarRocks: 1. Flink CDC + SMT (Recommended for Real-time) This is the most "official" and high-performance way to replicate data from databases like MySQL, PostgreSQL, Oracle, and SQL Server. * How it works: You use the StarRocks Migration Tool (SMT) to automatically scan your source database. It generates the

CREATE TABLE

statements for StarRocks and the Flink SQL jobs for you. * Why it's better: It maps source columns to StarRocks columns directly. It also supports schema evolution and can handle deletes/updates if you use the Primary Key model in StarRocks. * Best for: Real-time, sub-second latency replication. 2. CloudCanal CloudCanal is a commercial data synchronization platform (with a community edition) that has native, high-quality support for StarRocks. * How it works: It provides a visual web interface where you select your source (e.g., MySQL) and destination (StarRocks). It handles schema migration, full data seeding, and incremental synchronization automatically. * Why it's better: It is "no-code" and very user-friendly. It handles the data parsing and mapping for you without needing to write JSON parsing logic. 3. Apache SeaTunnel SeaTunnel is a very popular next-generation data integration tool designed for high-performance synchronization. * How it works: You define a configuration file (Source -> Transform -> Sink). Its StarRocks Sink is highly optimized and can handle massive throughput. * Why it's better: It supports a huge variety of sources and allows you to perform basic transformations (like column renaming or type casting) in the pipeline so data arrives in StarRocks exactly how you want it. 4. DataX (with StarRocks Writer) DataX is a mature offline data synchronization tool (originally from Alibaba). * How it works: It uses a reader/writer architecture. You use the

starrockswriter

plugin. * Why it's better: It is excellent for batch/offline migrations from relational databases or HDFS. It maps source fields to StarRocks columns based on a JSON configuration file you provide. 5. StarRocks Kafka Connector If your data is already in Kafka (e.g., via Debezium), use the official StarRocks Kafka Connector. * How it works: It acts as a Kafka Connect Sink. * Why it's better: Unlike basic Airbyte loads, this connector can be configured to "unwrap" Debezium CDC records and map them directly to structured columns in a Primary Key table. Summary Comparison Tool | Type | Setup Complexity | Best Use Case Flink CDC + SMT | Real-time | Medium (Requires Flink) | Production-grade real-time sync with schema mapping. CloudCanal | Real-time/Batch | Low (Visual UI) | Users who want a GUI and "hands-off" schema migration. SeaTunnel | Real-time/Batch | Low/Medium | High-throughput, many-to-many data integration. DataX | Batch | Low | One-time or scheduled batch migrations. Rocky's Tip: If you want the least amount of manual table creation, start with SMT (StarRocks Migration Tool). It will save you hours by generating all the SQL you need to create your structured tables! References • loading/Loading_intro.md • loading/Kafka-connector-starrocks.md • loading/Spark-connector-starrocks.md

@Rocky for more info, the source we used are snowflake and postgres

Thanks for the info! Since you're using Snowflake and PostgreSQL, StarRocks provides several ways to integrate with these sources, ranging from direct federated queries to high-performance batch loading and real-time CDC. Here is a breakdown of the best approaches for each: 1. PostgreSQL Integration PostgreSQL is a very common source for StarRocks, and you have three main paths depending on your performance and latency needs: * JDBC Catalog (Easiest for Queries/Small Batches): You can create a JDBC Catalog to query PostgreSQL directly without moving data. This is great for small datasets or for doing a one-time migration using

INSERT INTO ... SELECT

.

sql
    CREATE EXTERNAL CATALOG pg_catalog
    PROPERTIES (
        "type" = "jdbc",
        "user" = "your_user",
        "password" = "your_password",
        "jdbc_uri" = "jdbc:postgresql://<host>:<port>/<database>",
        "driver_url" = "<https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar>",
        "driver_class" = "org.postgresql.Driver"
    );

* StarRocks Migration Tool (SMT) + Flink CDC (Best for Real-time): If you need sub-second synchronization, use SMT. It automatically generates the table schemas in StarRocks and the Flink CDC jobs to sync your Postgres WAL (Write Ahead Log) directly into StarRocks. * Kafka Connector: Use Debezium to capture changes from PostgreSQL and stream them into Kafka, then use the StarRocks Kafka Connector to sink them into your cluster. ────────── 2. Snowflake Integration Since Snowflake is an OLAP system, the integration usually focuses on large-scale data movement. * Broker Load via S3 (Best for Large Volumes): The most performant way to move data from Snowflake to StarRocks is to unload the data to cloud storage (like AWS S3) first. 1. In Snowflake, use

COPY INTO '<s3://your-bucket/path/>'

to export your data as Parquet or CSV. 2. In StarRocks, use a Broker Load to pull that data in:

sql
        LOAD LABEL snowflake_import
        (
            DATA INFILE("<s3a://your-bucket/path/*>")
            INTO TABLE target_table
            FORMAT AS "parquet"
        )
        WITH BROKER
        (
            "aws.s3.access_key" = "...",
            "aws.s3.secret_key" = "...",
            "aws.s3.region" = "us-east-1"
        );

* JDBC Catalog (For Direct Querying): Similar to PostgreSQL, you can use the JDBC Catalog to query Snowflake directly. This is useful for "joining" your Snowflake data with StarRocks data in real-time, though for massive tables, the S3 route above is generally faster. Which one should you start with? * If you just want to explore the data without moving it: Start with JDBC Catalogs. * If you are doing a large migration: Use Snowflake → S3 → StarRocks and SMT for Postgres. Do you have a preference for real-time vs. batch loading for either of these? I can help with the specific SQL syntax for your setup! References • loading/s3.md • loading/Loading_data_template.md • loading/BrokerLoad.md