1. @Rocky Routine Loads stay paused after a BE is lost When a BE goes down or restarts, our Routine Loads pause and stay paused until someone runs RESUME ROUTINE LOAD. From the FE source (ScheduleRule.isNeedAutoSchedule), a paused job is auto-resumed only if all of these hold: the pause reason is REPLICA_FEW_ERR; no other reason qualifies; the number of dead BEs is no more than max_tolerable_backend_down_num, which defaults to 0; the job has had fewer than 3 auto-resumes within period_of_auto_resume_min (default 5 min). After the third, autoResumeLock is set and only a manual resume brings the job back. So with 3 replicas and one BE down, data could still be written, but the job never resumes by itself. And once the BE is back, a job that hit the lock or paused for another reason still needs a manual resume. Questions: Is there a supported way to have Routine Loads auto-resume after transient BE loss, for other pause reasons too and without the permanent lock? For example, a per-job property, or retry with backoff. Is it safe to raise max_tolerable_backend_down_num above 0 when the replication factor is 3? Are there plans to make the resume count and lock configurable? What do you recommend for keeping Routine Loads running unattended in production in case of broker blips or intermittent infra connecvitiny issue with kafka?

The behavior you observed in the StarRocks source code is correct: Routine Load auto-resumption is intentionally conservative to prevent "retry storms" when a cluster is unstable. Here are the answers to your questions based on StarRocks documentation and best practices. 1. Auto-Resume for Other Reasons or Without the Lock Currently, there is no per-job property to bypass the "3 retries in 5 minutes" limit or to expand auto-resumption to all pause reasons. The

autoResumeLock

and the restriction to

REPLICA_FEW_ERR

are hardcoded safety mechanisms. However, you can influence the behavior via global FE configurations: *

max_tolerable_backend_down_num

: Increasing this (default

0

) allows the FE to attempt auto-resumption even if some BEs are offline. *

period_of_auto_resume_min

: This defines the interval (default

5

minutes) at which the FE checks for jobs that need rescheduling. 2. Safety of Raising

max_tolerable_backend_down_num

It is generally safe to raise

max_tolerable_backend_down_num

to 1 if your replication factor is 3. * Quorum Integrity: With 3 replicas, StarRocks only needs a majority (2 replicas) to remain available for writes to succeed. Losing one BE does not jeopardize data consistency or availability. * The "Lock" configuration: There are currently no public configuration parameters to change the "3-retry" limit or the specific

autoResumeLock

behavior. These are internal FE scheduler rules designed to protect the cluster from infinite loops on permanent failures (like table deletion or schema mismatch). 3. Recommendations for Unattended Production Operation To keep Routine Loads running through intermittent infrastructure or Kafka blips, consider the following configuration strategy: FE Configurations * Set

max_tolerable_backend_down_num = 1

(or higher depending on cluster size). This ensures that a single BE restart doesn't permanently stall your ingestion. * Monitor

starrocks_fe_routine_load_paused

: Use this Prometheus metric to trigger alerts. Since the system has a hard stop after 3 failed auto-resumes, external monitoring is required for high-availability production environments. Job-Specific Properties *

max_error_number

/

max_filter_ratio

: Avoid leaving

max_error_number

at

0

. If Kafka occasionally receives a malformed message, a setting of

0

will pause the job immediately. Set a threshold that reflects your acceptable data quality loss. *

pause_on_fatal_parse_error

: If you are on v3.3.12+ or v3.4.2+, ensure this is

false

(default) if you prefer to skip unparseable messages rather than pausing the job. Connectivity Settings *

routine_load_kafka_timeout_second

: (Available since v3.2.0) If you experience network "blips" or slow Kafka brokers, increase this BE configuration (default

10s

) to prevent tasks from timing out and triggering a pause. Automation For a truly "unattended" setup, many production users employ a sidecar script or a cron job that periodically runs:

sql
-- Resumes all paused jobs in a database
RESUME ROUTINE LOAD ALL;

This bypasses the internal

autoResumeLock

and is the standard way to handle jobs that have exhausted their internal retry budget due to transient environmental issues. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md