@Rocky our routine loads failed because volumes ran out of space. I expanded the volumes but the loads still timed out. I created new loads from offsets and they ran a while but they have timed out again. All backends heartbeat fine. Any ideas?

It is common for Routine Loads to experience timeouts when recovering from a disk-full event, even after volumes are expanded. This is usually due to a massive backlog of data in Kafka that causes individual tasks to exceed their processing or commit time limits. 1. Check the specific Error Reason First, identify why the loads are failing by running:

sql
SHOW ROUTINE LOAD FOR <job_name>\G

Look at the

ReasonOfStateChanged

and

ErrorLogUrls

fields. If the error is "TIMEOUT", it typically means the task was scheduled but failed to finish within the

routine_load_task_timeout_second

. 2. Verify Backend Disk Status Even if heartbeats are fine, ensure the FE has updated the disk usage information. By default, BEs report disk state every 60 seconds.

sql
SHOW BACKENDS\G

Check if

Avail

and

Total

reflect your expansion and that

IsDecommissioned

or

DiskUsed

isn't triggering high-watermark thresholds (which might still restrict writes). 3. Tune for Backlog Recovery When you restart a load from an old offset, the task tries to consume a large amount of data. If the processing (consumption + RPC to BE + tablet commit) takes longer than 60 seconds, it will timeout. Try increasing the task timeout and batch size: * Increase Task Timeout: Give the tasks more time to process the backlog.

sql
    ALTER ROUTINE LOAD FOR <job_name>
    PROPERTIES (
        "routine_load_task_timeout_second" = "300"
    );

* Increase Batch Size: Allow tasks to take more data per commit to reduce the frequency of metadata operations.

sql
    -- Increase the number of messages or data size per task
    ALTER ROUTINE LOAD FOR <job_name>
    PROPERTIES (
        "max_routine_load_batch_size" = "2000000",
        "routine_load_task_consume_second" = "30"
    );

4. Check for Compaction Backlog Running out of disk space often halts compaction. Once space is freed, the BEs may start "aggressive" compaction to merge the many small rowsets created before the failure. This creates high I/O and CPU pressure, which can slow down new load commits. * Check compaction status:

curl http://<be_ip>:<be_http_port>/get_var?var=compaction_mem_current_consumption

or check the

<http://be.INFO|be.INFO>

logs for compaction activity. 5. Inspect BE Logs Search the

<http://be.INFO|be.INFO>

logs on the nodes where the tasks are running for "routine load" or the specific job ID. Specifically, look for: *

left_bytes

/

left_time

: If

left_bytes

is less than 0, the task is hitting the batch size limit before the time limit, indicating it's trying to catch up. *

[E1008]Reached timeout

: This confirms the RPC between FE and BE timed out. Recommendation: If the loads continue to fail after increasing the

task_timeout_second

to 300+, check the BE logs to see if there are underlying tablet issues (e.g., "too many versions" or "replica not found") that may have been triggered by the previous disk-full state. References • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • administration/management/monitoring/Monitor_and_Alert.md

@Rocky I set property below, but timeout still happened at 30s

ALTER ROUTINE LOAD FOR my_load PROPERTIES ("task_timeout_second" = "120");

The reason your Routine Load tasks are still stopping (or "timing out") at 30 seconds after you set

task_timeout_second

to 120 is likely due to the auto-calculation of the consumption window. In StarRocks (starting from v3.1), if you only specify one of the timeout properties, the other is automatically derived using a 1:4 ratio: *

task_consume_second

: The time the task spends actually reading from Kafka. *

task_timeout_second

: The total allowed time for the entire task (consumption + data processing + commit). * The Rule: If you only set

task_timeout_second

, then

task_consume_second

is automatically set to

task_timeout_second / 4

. In your case,

120 / 4 = 30 seconds

. This explains why the task stops its consumption phase exactly at 30 seconds. To allow the load to run longer and process more data per task (which is critical when recovering from a disk-full backlog), you should explicitly set both properties: 1. Update both properties Run the following to give the task a longer consumption window:

sql
ALTER ROUTINE LOAD FOR my_load 
PROPERTIES (
    "task_consume_second" = "60",
    "task_timeout_second" = "180"
);

(A ratio of 1:3 or 1:2 is often better when the cluster is under heavy I/O load, as it leaves more time for the "commit" phase.) 2. Check for "Too Many Versions" Since your disks were full, compaction likely fell behind. When you resumed the loads, the backlogs created many small rowsets. Check the

ReasonOfStateChanged

in `SHOW ROUTINE LOAD`: * If you see "too many versions", do not just increase timeouts. You need to slow down the load to let compaction catch up. * To do this, increase the batch size so you create fewer, larger rowsets:

sql
    ALTER ROUTINE LOAD FOR my_load
    PROPERTIES (
        "max_routine_load_batch_size" = "104857600", -- 100MB
        "task_consume_second" = "60",
        "task_timeout_second" = "120"
    );

3. Verify the Property via Task Monitoring To confirm what timeout the tasks are actually running with, check the active tasks:

sql
SHOW ROUTINE LOAD TASK WHERE JobName = "my_load"\G

Look at the

Timeout

column in the output. This shows the effective timeout being used by the FE for currently scheduled tasks. 4. BE Configuration (Version 3.2+) If you are on StarRocks 3.2 or later and still see RPC-level timeouts, check the BE configuration

routine_load_kafka_timeout_second

(default 10s). If your Kafka brokers are sluggish due to the massive request volume, you may need to increase this in

be.conf

. Summary: The 30s limit you are seeing is almost certainly the derived

task_consume_second

. Set both

task_consume_second

and

task_timeout_second

explicitly to bypass the

ALTER

statement. References • administration/management/BE_configuration.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • administration/management/FE_configuration.md