@Rocky Context: We're running StarRocks 4.1.0 FE / 4.0.9 CN, 3 FE replicas, with Query Queue v1 (enable_query_queue_select: true, query_queue_concurrency_limit: 32, query_queue_pending_timeout_second: 15). Per the docs from the Leader FE collectively manages the running-query count, with Followers notifying it on query start/finish. Observed behavior: During a rolling restart of our FE StatefulSet, we saw near-100% query failures (Failed to allocate resource to query: pending timeout [15s]) for ~2 minutes — a window that correlates closely with two back-to-back Leader elections (the Leader was restarted, its successor was restarted again shortly after). Critically, this failure rate was observed even on the two FE pods that were not being restarted and were otherwise healthy. Our working hypothesis: the concurrency check on a Follower requires successfully coordinating with the Leader to confirm/update the running-query count. When the Leader is mid-restart or mid-election, that coordination can't complete, so the Follower's admission check stalls (fails closed) rather than admitting the query — even though the Follower itself is fully healthy and the query itself would run fine. Questions: 1. Is this correct? When a Follower FE cannot reach the current Leader (mid-election or Leader is down), does its query-queue admission check block/queue the query until the Leader becomes reachable again (fail-closed), rather than admitting it optimistically (fail-open) or falling back to a locally-cached count? 2. Is there a supported way to avoid this — e.g., a timeout/fallback specifically for "Leader unreachable" distinct from the general pending-queue timeout, or a recommended restart/leader-transfer sequencing for rolling FE upgrades that avoids this window entirely? 3. Is this considered expected behavior for Query Queue v1, or would it be worth filing as a robustness gap? (We're also evaluating Query Queue v2 as an alternative — does v2 change this Leader-dependency for the concurrency check?)

Hello! I am Rocky, the StarRocks chatbot. Your hypothesis is correct. In StarRocks versions 3.1.4 and later (including your v4.x setup), Query Queue v1 implements Collective Concurrency Management, which relies on the Leader FE as the central authority for the cluster-wide running-query count. 1. Leader Dependency in Query Queue v1 When

enable_query_queue_select

is enabled in v3.1.4+, Follower FEs must notify the Leader FE via RPC whenever they initiate or finish a query to update the global concurrency count. * Behavior during Election: If the Leader FE is unreachable (restarting or mid-election), the Follower's admission check cannot complete. * Fail-Closed: The system is designed to "fail-closed" to prevent concurrency spikes that could crash BEs. Instead of admitting the query and risking an overload, the Follower stalls the query, placing it in the

PENDING

state. * Timeout: If the Leader election takes longer than your

query_queue_pending_timeout_second

(15s in your case), those pending queries will time out and fail with the error:

Failed to allocate resource to query: pending timeout

. 2. Mitigation and Best Practices Currently, there is no "fail-open" or "locally-cached" fallback for the global concurrency check, as the primary goal of the feature is strict resource protection. To minimize this impact: * Adjust Timeout: Increase

query_queue_pending_timeout_second

to a value higher than your typical FE election window (e.g., 30–60 seconds). This allows queries to "wait out" the election rather than failing immediately. * Restart Sequencing: During a rolling restart, the Leader FE should be the last pod to be restarted. This ensures that a Leader is available for the maximum amount of time for the other FEs. * Leader Transfer: While StarRocks automatically elects a new leader when the current one stops, the "double election" you saw (Leader restarted, then successor restarted) is particularly disruptive. Ensure your StatefulSet

terminationGracePeriodSeconds

and readiness probes are configured to allow the new Leader to fully synchronize metadata before the next pod is taken down. 3. Query Queue v2 vs. v1 Query Queue v2 (introduced in v3.3/v3.4 and enabled by default from v4.1) introduces a more sophisticated "slot-based" scheduling system, but the core architectural dependency remains similar: * Centralized Slot Management: In v2, the Global Slot Manager also resides on the Leader FE. Follower FEs still need to request slots from the Leader to execute queries. * Difference: V2 is better at estimating resources (CPU/Memory) and uses different scheduling strategies (like SWRR or SJF), but it does not remove the Leader as the source of truth for global resource allocation. * Recommendation: You should definitely evaluate Query Queue v2. Since you are on v4.1, v2 is likely already the intended default. It provides much better granularity than v1’s simple "concurrency count" by calculating "slots" based on query complexity and available BE resources. Summary The behavior you observed is expected for Query Queue v1 in its current collective management implementation. It is considered a trade-off for strict resource isolation. To improve robustness, I recommend increasing the pending timeout to cover the election window and migrating to Query Queue v2 for better resource management. For further verification or to report this as a robustness gap (e.g., requesting an "optimistic admission" fallback), I suggest reaching out on the StarRocks Community Slack or filing an issue on the **StarRocks GitHub repository. References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • faq/resource_isolation_faq.md

@Rocky Appreciate the confirmation on fail-closed design. None of the three suggestions actually work for us: 1. Raising the timeout — already tried, at 300s (before a 2026-07-02 fix cut it to 15s), and it still failed. That incident was an 8-minute kube-dns outage (both DNS pods evicted, no PDB) breaking FE-FE coordination — longer than any timeout we'd accept. We fixed it by hardening DNS and cutting the timeout to fail fast, not by extending it. So this isn't limited to leader restarts — any coordination gap (DNS, network, GC) hits the same wall, and no timeout survives all of them without blowing our sub-1s SLA. 2. Restart Leader last — cuts down double-elections (we saw two back-to-back on 2026-07-07: ~27s + ~38s gaps), but doesn't remove the one gap every leader restart still needs. 3. Query Queue v2 — same Leader dependency for slot management, per your answer. Doesn't fix this. Question: for planned restarts, can Leader be transferred to a healthy Follower before the current Leader stops — so the new Leader is confirmed active with no gap — instead of stop-then-elect?

That is a sharp observation. You’ve correctly identified the "wall" created by the collective management design: because the leader is the sole source of truth for the global concurrency count, its absence (or unreachable state) creates a hard block for all followers. 1. Pre-emptive Leader Transfer Unfortunately, there is no supported SQL command or API (e.g.,

ALTER SYSTEM TRANSFER LEADER

) to trigger a "handoff" where a new leader is confirmed active before the current leader relinquishes its role. In StarRocks (and the underlying BDBJE replication), leadership transition is a stop-then-elect process. Even with the Graceful Exit feature (

stop_fe.sh -g

) introduced in v3.3+, the Leader FE's procedure is to close its metadata store (BDBJE) and then "allow a new Leader election to complete." While this is cleaner than a hard

SIGKILL

, it still results in a window (typically 5–30 seconds) where no leader is available to authorize query slots. 2. Recommended Workaround for Planned Restarts Since your goal is to avoid the "fail-closed" failures during a sub-1s SLA window, the most effective strategy for planned maintenance (like a rolling upgrade) is to temporarily bypass the leader-dependent check: 1. Disable the Query Queue globally before restarting the Leader FE:

sql
    SET GLOBAL enable_query_queue_select = false;

This stops Follower FEs from trying to reach the leader for concurrency checks. Queries will be admitted immediately without queuing. 2. Perform the Leader restart/upgrade. 3. Wait for the new Leader to be stable (verify via

SHOW FRONTENDS

). 4. Re-enable the Query Queue:

sql
    SET GLOBAL enable_query_queue_select = true;

This effectively shifts the system from fail-closed to temporarily open during the maintenance window. While this technically risks a concurrency spike, it is usually preferable to the 100% failure rate caused by the election gap. 3. Query Queue v2 and Robustness * Leader Dependency: You are correct that Query Queue v2 does not solve this specific problem. It still centralizes "Global Slot Management" on the Leader FE. * Robustness Gap: This is indeed a known architectural trade-off. Collective management was designed to solve "over-admission" (where BEs were overwhelmed because multiple FEs didn't know about each other's queries), but it introduced this leader-side single point of failure. * Filing a Gap: It is worth filing this as a robustness gap on GitHub. Potential improvements the community has discussed in the past (but not yet implemented) include: * Optimistic Admission: Allowing followers to admit queries based on local/cached counts if the leader is unreachable. * Pre-emptive Transfer: Exposing the BDBJE

transferMaster

call to allow a zero-gap handover. Summary: For your v4.1 environment, the most reliable path for planned restarts is the Disable -> Restart Leader -> Enable sequence. For unplanned outages (like your DNS incident), the fail-closed behavior remains the default protection mechanism. References • administration/management/graceful_exit.md • administration/management/resource_management/query_queues.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md References • administration/management/resource_management/query_queues.md

@Rocky so disabling the query queue totally would also help us here? Because then we will go with that for now.

Yes, disabling the query queue entirely will resolve the specific failure you are seeing. By setting

enable_query_queue_select = false

, you remove the requirement for Follower FEs to check in with the Leader FE before admitting a query. This effectively moves your cluster from a fail-closed (protected but fragile) state to a fail-open (resilient but unprotected) state. What happens when you disable it: 1. Immediate Admission: When a query hits a Follower FE, it will be admitted immediately and sent to the BEs/CNs for execution without any RPC call to the Leader for concurrency "slots." 2. Election/Outage Resilience: If the Leader is restarting, mid-election, or unreachable due to DNS issues, the Followers will continue to process queries normally. Your sub-1s SLA will no longer be threatened by the "coordination wall." 3. No More "Pending Timeout": Since there is no queue, queries will never enter the

PENDING

state and therefore cannot time out waiting for resource allocation. The Trade-off (The "Risk"): The Query Queue exists to prevent cluster-wide resource exhaustion. If you disable it: * Concurrency Spikes: If your application suddenly sends 500 complex queries at once, the cluster will attempt to run all of them simultaneously. This could lead to BE/CN memory exhaustion (OOM) or extremely high CPU wait times, which might cause a different kind of SLA breach. * Loss of Global Guardrails: You lose the ability to enforce the

query_queue_concurrency_limit: 32

across the whole cluster. Recommendation for v4.1.0 Since you are on v4.1.0, you should be aware that Query Queue v2 is enabled by default (

enable_query_queue_v2 = true

). * If you want to ensure all centralized queuing is disabled to avoid any Leader-dependency, you should set:

sql
    SET GLOBAL enable_query_queue_select = false; -- Disables v1 Select Queue
    SET GLOBAL enable_query_queue = false;        -- Disables v2 Global Queue

* Alternatively, you can keep the Query Queue enabled for standard operations and only disable it during your rolling restart maintenance windows (as suggested in the previous response). This gives you protection during normal peaks but avoids the "suicide" behavior during planned leader elections. In summary: If your priority is ensuring queries never fail due to Leader FE unavailability/elections, and you are confident your BE/CN nodes can handle your peak query load without a hard concurrency cap, disabling the queue is the correct path forward. References • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • sql-reference/System_variable.md References • administration/Meta_recovery.md