🧵 rack aware routine load ingestion in starrocks

I'm here to help! Please let me know what issue you're facing or what question you have about StarRocks. Once you provide the details (such as error messages, queries, or the specific behavior you're seeing), I can search the documentation and resources for you. If it's a complex issue I can't resolve, I can then suggest the best channels for reaching out to the community or maintainers.

check. my above messages @Rocky

I don't see any specific technical details or questions in our conversation yet. Could you please paste the error message, the query you're running, or describe the specific behavior you're seeing? Once you provide that information, I can search the documentation and help you troubleshoot.

@Rocky rack aware routine load ingestion in starrocks 5 replies Ayush [2:49 PM] can anyone help how to read rack aware routine load kafka data in starrocks, provided routine load will run again and again, it can run on any be node, so that be node should create rack aware consumer connection with kafka [2:49 PM] @Rocky can you help or tag someone who can help with this Rocky [2:49 PM] I'm here to help! Please let me know what issue you're facing or what question you have about StarRocks. Once you provide the details (such as error messages, queries, or the specific behavior you're seeing), I can search the documentation and resources for you. If it's a complex issue I can't resolve, I can then suggest the best channels for reaching out to the community or maintainers.

To achieve rack-aware ingestion with Routine Load in StarRocks (specifically for Kafka "Follower Fetching"), you need to pass the

client.rack

property to the underlying

librdkafka

consumer. 1. How to set

client.rack

You can include any Kafka consumer property in your

CREATE ROUTINE LOAD

statement by prefixing it with

property.

. For rack awareness, you would add:

sql
CREATE ROUTINE LOAD your_db.your_job ON your_table
PROPERTIES (
    "format" = "json"
)
FROM KAFKA (
    "kafka_broker_list" = "broker1:9092,broker2:9092",
    "kafka_topic" = "your_topic",
    "property.client.rack" = "rack-id-1" -- Replace with your rack ID
);

2. The Challenge: Dynamic Rack Assignment The issue you mentioned is that Routine Load tasks are scheduled across any available BE node. Because the

client.rack

property is part of the job definition stored in the FE (Frontend), it is currently static. * Current Limitation: StarRocks does not yet have a built-in feature to automatically "inject" the current BE's rack ID (from

be.conf

or BE labels) into the Routine Load task at runtime. * Static Mapping: If you set

"property.client.rack" = "rack-1"

, all tasks for that job will report as being in

rack-1

to Kafka, even if they are running on a BE physically located in

rack-2

. 3. Possible Workarounds If you must ensure that consumers only fetch from the same rack to save cross-AZ traffic costs: * Separate Jobs per Rack: Create separate Routine Load jobs for each rack. However, StarRocks does not natively support "pinning" a Routine Load job to a specific group of BEs based on rack labels via the

CREATE ROUTINE LOAD

syntax. * BE-level global config: There is currently no global

be.conf

parameter to set default

librdkafka

properties (like

client.rack

) for all Routine Load tasks running on that specific BE. 4. Recommendation Since you are asking about a scenario where the task "can run on any BE node," and you want the BE to automatically identify its rack to Kafka: 1. Verify your StarRocks version: Ensure you are on a recent version (v3.2+), as location-awareness features are being actively expanded. 2. Community Inquiry: Because this specific "Dynamic Rack-Aware Ingestion" requirement is not explicitly covered in the current public documentation, I recommend reaching out to the StarRocks community Slack or opening a GitHub issue. This would likely require an enhancement where the BE process automatically appends its

labels.location

(if set) to the Kafka consumer's

client.rack

property. If you would like me to tag a maintainer or search for a specific PR regarding this, let me know! References • administration/management/BE_configuration.md

"property.client.rack" = "rack-id-1"

this will run on one node, now second time routine load can get schedule on another be node, how it dynamically resolve @Rocky

Currently, StarRocks does not have a built-in feature to dynamically resolve the

client.rack

property for Routine Load based on the BE node's physical rack. When you set

"property.client.rack" = "rack-1"

in a

CREATE ROUTINE LOAD

statement, that value is static and is stored in the job's metadata on the FE. Since the FE schedules Routine Load tasks across all available BE nodes to balance the load, every task—regardless of which BE executes it—will report its rack as

"rack-1"

to Kafka. Why this is a challenge For Kafka "Follower Fetching" (rack-aware consumption) to work effectively: 1. The Kafka consumer's

client.rack

must match the

broker.rack

of the brokers in its same availability zone/rack. 2. If a Routine Load task is scheduled on a BE in Rack B, but the job property is statically set to Rack A, the consumer will incorrectly attempt to fetch from Rack A brokers, potentially increasing cross-AZ network traffic and costs. Current Status and Workarounds As of the current version, there is no automatic mapping between a BE's

location

label (set in

be.conf

) and the Kafka

client.rack

property in Routine Load. If you must implement rack-aware ingestion today, you may consider these limited options: 1. Multiple Routine Load Jobs (Manual Partitioning): If your Kafka topic is partitioned and you can identify which BEs should handle which partitions, you could theoretically create multiple Routine Load jobs. However, StarRocks does not yet support "pinning" Routine Load tasks to specific BE groups (Resource Groups) based on rack labels, so the FE may still schedule tasks for "Job-Rack-A" on a BE in "Rack-B." 2. External Kafka Proxy: Some users deploy a local Kafka proxy (like

envoy

or a custom sidecar) on each BE node. The BE is configured to connect to

localhost:port

, and the proxy injects the correct rack ID before forwarding the request to the Kafka cluster. This is an advanced architectural workaround and not a native StarRocks feature. 3. Community Feature Request: This specific requirement—dynamic injection of BE location labels into the Kafka consumer config—is a valid feature request for the StarRocks roadmap. I recommend checking the StarRocks GitHub Issues or bringing this up in the

#support

channel of the StarRocks Slack to see if it is planned for a future release (e.g., v3.3+). Summary: The property is currently static. There is no native way to make it dynamic per BE node without manual intervention or architectural workarounds. References • administration/management/BE_configuration.md