Hey! I am just wondering if there are any plans to expand the write support via Flight SQL, in particular implementing the bulk ingestion imports so that we can write fast using polars/pandas. With Postgres for example FlightSQL writes are 100x faster (~1M/rows/sec) vs current StarRocks 10k/rows/sec using SQLAlchemy executemany(). I understand that FILE() and object storage based imports are the way to go for larger imports, but it requires a lot more coordination (ie. output file to parquet, upload to object storage, then run insert query) would be great to just be able to send it out over the wire in a native arrow table. This is really useful for querying via Python where we sometimes create larger temporary tables to join against (large unloading). At the moment most of the time of these queries is loading the temporary table, the actual execution and retrieval (via FlightSQL) is super fast.

Hello, @Gus Minto-Cowcher! How can I assist you today?

@Rocky Hey! I am just wondering if there are any plans to expand the write support via Flight SQL, in particular implementing the bulk ingestion imports so that we can write fast using polars/pandas. With Postgres for example FlightSQL writes are 100x faster (~1M/rows/sec) vs current StarRocks 10k/rows/sec using SQLAlchemy executemany(). I understand that FILE() and object storage based imports are the way to go for larger imports, but it requires a lot more coordination (ie. output file to parquet, upload to object storage, then run insert query) would be great to just be able to send it out over the wire in a native arrow table. This is really useful for querying via Python where we sometimes create larger temporary tables to join against (large unloading). At the moment most of the time of these queries is loading the temporary table, the actual execution and retrieval (via FlightSQL) is super fast.

That is a great question! You are spot on about the performance bottleneck—row-based protocols like

SQLAlchemy.executemany()

or standard MySQL/PostgreSQL protocols are indeed limited by row-column conversions and serialization overhead, which usually caps ingestion at around 10k–20k rows/sec. The good news is that StarRocks has been actively working on this. Starting from StarRocks v3.5.1, a new high-performance data transfer channel based on the Apache Arrow Flight SQL protocol was introduced as an experimental feature. This is specifically designed to optimize both data retrieval (unloading) and data import (ingestion). Current State & Plans for Flight SQL Writes * Version 3.5.1+: Introduces the optimized import channel for Arrow Flight SQL. It aims to eliminate the row-column conversion overhead and achieve the "zero-copy" style throughput you're looking for. * Experimental Phase: While the protocol support is there in the engine, the high-level Python library support (like a native

adbc.ingest

that matches the 1M rows/sec performance of other systems) is still maturing. * Native Arrow Support: The goal is indeed to allow you to send an Arrow Table directly over the wire without staging it in object storage. How to get high-speed writes today? While you wait for the Flight SQL bulk-write client ecosystem to fully mature in Python, the recommended "native" way to get ~1M rows/sec from Python right now is Stream Load. You can avoid the "coordination" overhead (saving to S3/HDFS) by streaming the data directly from memory. For Polars/Pandas users, the most efficient pattern is to convert your dataframe to a Parquet buffer in memory and then POST it to the StarRocks Stream Load API. Here is a quick example of how to do this efficiently from Python:

python
import polars as pl
import requests
import io

def stream_load_polars(df: pl.DataFrame, host, port, db, table, user, password):
    # 1. Convert Polars DF to Parquet in-memory
    buf = io.BytesIO()
    df.write_parquet(buf)
    buf.seek(0)

    # 2. Use Stream Load API
    # StarRocks can ingest Parquet directly via HTTP
    url = f"http://{host}:{port}/api/{db}/{table}/_stream_load"
    
    headers = {
        "format": "parquet",
        "label": f"label_{int(time.time())}",
        "Expect": "100-continue"
    }
    
    auth = requests.auth.HTTPBasicAuth(user, password)
    
    # This sends the arrow-backed data directly over the wire
    response = requests.put(url, data=buf, headers=headers, auth=auth)
    return response.json()

Summary of what's coming: * StarRocks v3.5.x is the version where these Flight SQL import optimizations are landing. * The Arrow Flight SQL ADBC driver is the intended interface for this. As the driver evolves, you will eventually be able to use

adbc_driver_manager.ingest

with StarRocks at the speeds you observed with other Flight SQL implementations. If you are currently on an older version, upgrading to v3.5.1+ and keeping an eye on the

adbc-driver-flightsql

updates is the way to go! References • unloading/arrow_flight.md

@Rocky I am on 4.0.6, does stream load support parquet? Currently I can only use FlightSQL for fast unloading and executemany() style loading (no bulk ingestion) so it is still slow on the ingestion path.

Hello! I am Rocky, your StarRocks chatbot. Regarding your question about version 4.0.6 and Parquet support in Stream Load: As of the current stable releases (including the 3.x series), Stream Load (the synchronous HTTP-based ingestion method) natively supports CSV and JSON formats. Official documentation for the latest versions still lists Parquet support for Stream Load as "To be supported". If you are using a version labeled 4.0.6 (which might be a specific internal or upcoming build, as the current standard public releases are in the 3.x range), you should verify if

format=parquet

is accepted in the HTTP headers. However, for most users looking for high-performance bulk ingestion of Parquet data, the following methods are recommended: Recommended Bulk Ingestion for Parquet 1. INSERT from FILES (Recommended): If your Parquet files are stored in cloud storage (S3, GCS, OSS, Azure) or HDFS, you can use the

FILES()

table function. This is the modern, high-performance way to load Parquet files.

sql
    INSERT INTO your_table
    SELECT * FROM FILES(
        "path" = "<s3://your-bucket/data.parquet>",
        "format" = "parquet",
        "aws.s3.access_key" = "...",
        "aws.s3.secret_key" = "..."
    );

2. Broker Load: An asynchronous method that has supported Parquet for a long time. It is suitable for large-scale background data ingestion from remote storage or NAS. Speeding up the Ingestion Path from Applications If you are currently using

executemany()

via a database driver, you are likely experiencing a bottleneck because those methods typically translate to individual or small-batch

INSERT

statements, which are much slower than bulk loading. To significantly speed up your ingestion: * Switch to Stream Load with CSV: If your application can generate CSV data, pipe it to the Stream Load HTTP API. It is significantly faster than

executemany()

. * Compression: You can use

format=gzip

with Stream Load to reduce network overhead for large CSV/JSON payloads. * FlightSQL: While FlightSQL is excellent for fast unloading, for loading, Stream Load remains the primary high-throughput path for most application-to-StarRocks scenarios. If you specifically need to stream Parquet format over HTTP, please keep an eye on upcoming release notes for v3.4+ or v4.x, as expanding format support for Stream Load is a high-priority roadmap item. Let me know if you'd like more details on setting up

INSERT from FILES

or optimizing Stream Load! References • loading/loading_introduction/feature-support-loading-and-unloading.md • loading/StreamLoad.md • loading/hdfs_load.md • loading/Loading_data_template.md