@Rocky does starrocks pipe support Cloudflare R2 as a file source? Not documented here but wonder how much work is it to support it. docs.starrocks.io/docs/integrations/streaming/pipe

Yes, StarRocks Pipe supports Cloudflare R2 as a file source. While not explicitly listed by name in every Pipe-specific tutorial, StarRocks Pipe (introduced in v3.2) is built on top of the

FILES()

table function. Since

FILES()

supports generic S3-compatible storage, you can connect to Cloudflare R2 by providing the R2 S3 API endpoint and credentials. How to configure it To use Cloudflare R2 with StarRocks Pipe, you need to set the

aws.s3.endpoint

to your R2-specific URL. Example

CREATE PIPE

statement for R2:

sql
CREATE PIPE r2_pipe
PROPERTIES
(
    "AUTO_INGEST" = "TRUE"
)
AS
INSERT INTO your_table
SELECT * FROM FILES
(
    "path" = "<s3://your-bucket-name/data/*.parquet>",
    "format" = "parquet",
    -- R2 S3 API endpoint: https://<ACCOUNT_ID>.<http://r2.cloudflarestorage.com|r2.cloudflarestorage.com>
    "aws.s3.endpoint" = "https://<your_account_id>.<http://r2.cloudflarestorage.com|r2.cloudflarestorage.com>",
    "aws.s3.access_key" = "<your_r2_access_key>",
    "aws.s3.secret_key" = "<your_r2_secret_key>",
    "aws.s3.region" = "auto", -- R2 often uses 'auto'
    "aws.s3.enable_ssl" = "true"
);

Key Considerations * S3 Compatibility: Cloudflare R2 is S3-compatible. StarRocks interacts with it using the same logic it uses for AWS S3 or MinIO. * Authentication: Ensure your R2 API Token has the necessary permissions (Read/Write) for the bucket you are accessing. * Path Style: For some S3-compatible systems, you might need to toggle

"aws.s3.enable_path_style_access" = "true/false"

depending on how you've structured your endpoint, though R2 typically works with the standard virtual-hosted style provided by the

<account_id>

endpoint. * Efficiency: Because StarRocks Pipe uses

ETag

(standard for S3/R2) to track file uniqueness, it can efficiently perform incremental loads from R2 without re-processing old files. Cloudflare's own documentation also lists StarRocks as a compatible engine for their R2 Data Catalog (Iceberg), further confirming that the two systems are interoperable for data lakehouse and loading workflows. References • sql-reference/sql-functions/table-functions/files.md

@Rocky are you sure incremental load is supported? Once I use pipe to load into native table. Are you sure incremental materialized view is also supported?

Yes, both features are supported and work together effectively for a continuous data pipeline. 1. Are you sure incremental load is supported for Pipe? Yes. StarRocks Pipe is designed specifically for incremental loading from object storage (S3, R2, HDFS). * Uniqueness Tracking: Pipe tracks which files have already been loaded using a digest. For S3-compatible storage like Cloudflare R2, it uses the

ETag

of the file. * Continuous Ingestion: By setting

"AUTO_INGEST" = "TRUE"

, the Pipe job will periodically poll the source bucket/prefix. It compares the

ETag

and file name against its history (stored in

information_schema.pipe_files

) and only loads new or modified files. * Micro-batching: Unlike a standard

INSERT INTO SELECT FROM FILES()

, a Pipe breaks down a large set of files into smaller transactions, ensuring that if one file fails, the others still commit and are not re-processed. 2. Are incremental Materialized Views (MVs) supported on these tables? Yes. Since StarRocks Pipe loads data into native StarRocks tables, those tables can serve as the base for an Asynchronous Materialized View with incremental refresh. * How it works: When Pipe performs an

INSERT

into your native table, it creates a new data version (or updates a partition). The Asynchronous MV monitoring that table detects the version change. * Incremental Refresh Mode: You can define the MV with

PROPERTIES("refresh_mode" = "INCREMENTAL")

. * Partition-level Incremental: The most efficient pattern is to have both your native table and your MV partitioned (e.g., by date). When Pipe loads new data into a specific partition of the native table, StarRocks will only refresh the corresponding partition in the Materialized View. Example workflow: 1. Cloudflare R2: New Parquet files arrive in

<s3://my-bucket/events/dt=2023-10-01/>

. 2. StarRocks Pipe: Detects new ETags, executes micro-batch

INSERT

into native table

raw_events

. 3. Materialized View: Since

raw_events

was updated, the MV (if set to

ASYNC

) will trigger a refresh. If it's a partitioned MV, it will only re-calculate the

2023-10-01

partition. Summary of Requirements * StarRocks Version: v3.2 or higher (Pipe was introduced in 3.2). * Table Type: The destination of the Pipe must be a native StarRocks table (Duplicate, Unique, or Primary Key model). * MV Configuration: Use

REFRESH ASYNC

and, ideally,

PARTITION BY

to ensure the "incremental" behavior is efficient at scale. References • loading/s3.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md • sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md References • using_starrocks/async_mv/Materialized_view.md • sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md • loading/Loading_intro.md • loading/InsertInto.md