Throttling Routine Load ingestion rate — options other than a resource group? StarRocks 3.5.x, Routine Load from Kafka into a DUPLICATE KEY table When a backlog builds, the job drains it at full speed — we measured ~15K rows/s average and 68K peak — which spikes BE disk IO exactly when we'd rather it caught up gently. We can't pace it with max_batch_rows: it appears to be floored at 200,000, so setting it lower has no effect. max_batch_interval caps how long a task runs rather than its rate, and desired_concurrent_number is the only other lever we've found. The levers all bound batch size or duration, not the gap between tasks — so on a backlog it just runs back-to-back 200K batches. is there some similar way like we have in kafak via concurrency and other configs as fine control

Hello! I'm Rocky, your StarRocks assistant. I can help you with questions about StarRocks syntax, features, configuration, or troubleshooting. How can I help you today?

@Rocky Throttling Routine Load ingestion rate — options other than a resource group? StarRocks 3.5.x, Routine Load from Kafka into a DUPLICATE KEY table When a backlog builds, the job drains it at full speed — we measured ~15K rows/s average and 68K peak — which spikes BE disk IO exactly when we'd rather it caught up gently. We can't pace it with max_batch_rows: it appears to be floored at 200,000, so setting it lower has no effect. max_batch_interval caps how long a task runs rather than its rate, and desired_concurrent_number is the only other lever we've found. The levers all bound batch size or duration, not the gap between tasks — so on a backlog it just runs back-to-back 200K batches. is there some similar way like we have in kafak via concurrency and other configs as fine control

To throttle your Routine Load ingestion rate and force a "gentle catch-up" without using Resource Groups, you need to combine several job properties to reduce the "duty cycle" of the load tasks. As you observed,

max_batch_rows

is floored at 200,000 (to prevent excessive small-file versioning), so it cannot be used to shrink the batch size further. However, you can use the following levers: 1. Force Serial Consumption If your topic has multiple partitions, StarRocks will try to consume them in parallel up to your

desired_concurrent_number

. On a backlog, this results in multiple BEs hitting the disk simultaneously. * Action: Set

"desired_concurrent_number" = "1"

. This ensures only one task (consuming one or more partitions) runs at a time for that job, drastically reducing the aggregate IO spike. 2. Use

max_batch_size

to Bypass the Row Floor While

max_batch_rows

is floored at 200k, the

max_batch_size

(measured in bytes) is not. If 200k rows are too large for your disk to handle in a single burst, you can limit the task by byte size. * Action: Set

"max_batch_size" = "20971520"

(e.g., 20MB). The task will stop as soon as it hits 20MB, even if it hasn't reached 200,000 rows. 3. Maximize the Gap with

max_batch_interval

The

max_batch_interval

(default 10s) is the scheduling interval. In a backlog scenario, StarRocks tries to schedule the next task as soon as the previous one finishes. To force a gap: * Action: Set

"max_batch_interval" = "60"

. This is the maximum allowed value. * Note: If the task finishes in 5 seconds, StarRocks should wait for the remainder of the interval before scheduling the next one. If you still see "back-to-back" behavior, it is usually because

desired_concurrent_number

is > 1 or the FE

load_checker_interval_second

is triggering frequently. 4. Limit Consumption Time per Task Use

task_consume_second

(Job property, v3.1+) to tell the task exactly how long to pull data before committing. * Action: Set

"task_consume_second" = "5"

. * Interaction: If you set

task_consume_second = 5

and

max_batch_interval = 60

, the job will spend 5 seconds loading and then ideally "sleep" for the remainder of the 60-second window. 5. Kafka-Side Throttling (The "Kafka Way") Since Routine Load uses

librdkafka

under the hood, you can pass standard Kafka consumer properties in the

FROM KAFKA

clause to slow down the data retrieval rate: *

property.max.partition.fetch.bytes

: Limits how much data is fetched per partition per request. *

property.fetch.max.bytes

: Limits the total data fetched in one poll. *

property.fetch.message.max.bytes

: Limits individual message size. * Action: Add these to your

FROM KAFKA (...)

block to limit the network throughput, which indirectly smooths out the BE ingestion rate. Recommended Configuration for "Gentle Catch-up"

sql
ALTER ROUTINE LOAD FOR your_job_name
PROPERTIES (
    "desired_concurrent_number" = "1",  -- Stop parallel ingestion
    "max_batch_interval" = "60",        -- Maximize scheduling gap
    "task_consume_second" = "5",        -- Short work window
    "max_batch_size" = "52428800"       -- Cap at 50MB if 200k rows is too big
);

Why you see back-to-back tasks: During a backlog, StarRocks FE prioritizes clearing the lag. If

max_batch_interval

is set to the default (10s) and your task takes 8s to reach 200k rows, the 2s "gap" is barely noticeable. By increasing the interval to 60s and decreasing the work time to 5s, you force a ~90% idle time (duty cycle) per task. References • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • administration/management/monitoring/metrics.md