@Rocky would the pipe load work for cloudflare r2 the same way it works for aws s3?

Yes, StarRocks Pipe (introduced in v3.2) works for Cloudflare R2 just as it does for AWS S3, because R2 is fully S3-compatible. Since StarRocks Pipe uses the

FILES()

table function under the hood, you can configure it to point to your Cloudflare R2 bucket by providing the R2-specific endpoint and credentials. Key Differences in Configuration To use Pipe with Cloudflare R2, you need to adjust the following S3 properties in your

CREATE PIPE

statement: 1. Endpoint: Use your Cloudflare R2 endpoint (formatted as

https://<ACCOUNT_ID>.<http://r2.cloudflarestorage.com|r2.cloudflarestorage.com>

). 2. Path Style Access: Cloudflare R2 typically requires

aws.s3.enable_path_style_access = 'true'

. 3. Credentials: Use your R2 API Access Key and Secret Key. Example: Creating a Pipe for Cloudflare R2

sql
CREATE PIPE my_r2_pipe
PROPERTIES
(
    "AUTO_INGEST" = "TRUE",
    "POLL_INTERVAL" = "60" -- Polls R2 every 60 seconds for new files
)
AS
INSERT INTO my_target_table
SELECT * FROM FILES
(
    "path" = "<s3://my-bucket/data-folder/*>",
    "format" = "parquet",
    "aws.s3.endpoint" = "https://<YOUR_ACCOUNT_ID>.<http://r2.cloudflarestorage.com|r2.cloudflarestorage.com>",
    "aws.s3.access_key" = "<YOUR_R2_ACCESS_KEY>",
    "aws.s3.secret_key" = "<YOUR_R2_SECRET_KEY>",
    "aws.s3.enable_path_style_access" = "true",
    "aws.s3.region" = "auto" -- R2 uses 'auto' or doesn't strictly require a specific AWS region
);

Important Considerations * Polling vs. Event Notifications: Unlike some cloud-native loaders that rely on SQS (AWS) or Pub/Sub (GCS), StarRocks Pipe uses a polling mechanism (

AUTO_INGEST = TRUE

) to detect new files. This makes it compatible with any S3-compatible storage like R2 without needing external message queues. * File Uniqueness: Pipe uses the

ETag

of the object to ensure files aren't loaded twice. Cloudflare R2 provides

ETags

that StarRocks uses for this idempotency check. * Supported Formats: As with AWS S3, Pipe currently supports Parquet, ORC, and CSV (from v3.3) file formats. * Performance: Since R2 is often used with Cloudflare’s global network, ensure your StarRocks cluster has a stable and high-bandwidth connection to the Cloudflare endpoint to maintain high-speed ingestion speeds. References • loading/s3.md • loading/minio.md • loading/hdfs_load.md