@Rocky StarRocks autoscaling gap — CNs added mid-query can't help the query that triggered them Found while benchmarking on our shared-data cluster: when a heavy query saturates the CNs and the autoscaler adds more, the new nodes sit idle — StarRocks assigns work at query start and can't hand any to nodes that arrive later. The idle nodes then drag the average CPU down, so the autoscaler stops scaling. Net effect: heavy queries run on 1-2 nodes no matter how much capacity we add (measured ~1.7× slower than the same query with capacity available up front). Filed upstream: github.com/StarRocks/starrocks/issues/76485. Would like the team's take on: 1. pin CN count during heavy-query windows (simple, some idle cost) 2. Something else?

As you've correctly identified, StarRocks (and most MPP engines) generates and distributes query plan fragments to the available backends (CNs) at the start of execution. Currently, StarRocks does not support dynamic fragment migration or re-sharding for an active query. Since the autoscaler operates on cluster-wide averages, those new, idle nodes indeed "dilute" the average CPU metric, leading to premature scaling plateaus. Here is the "team take" on your proposed solutions and some additional strategies: 1. Pin CN count (Scheduled Scaling) This is currently the most reliable workaround if your heavy workloads are predictable (e.g., daily ETL or reporting windows). * How: If you are using the StarRocks Kubernetes Operator, you can use a CronJob to update the

minReplicas

of your

StarRocksCluster

spec or the HPA resource before the window starts. * Trade-off: You pay for the idle capacity during the ramp-up and ramp-down, but you eliminate the 1.7× latency penalty. 2. "Something Else?" (Alternative Mitigations) A. Query Queuing (Recommended) Instead of letting a massive query saturate the cluster and trigger a slow scale-out, use Query Queuing. * Benefit: It prevents "heavy query saturation" by holding incoming queries in a queue if the cluster is already at its resource limit. This ensures that the queries that are running have enough resources to finish quickly, and new queries will start only when the scaled-out nodes are actually ready and visible to the FE for the next plan. * Config: Set

enable_query_queue = true

and configure

query_queue_concurrency_limit

. B. Resource Isolation (Resource Groups) You can categorize "heavy" vs. "light" workloads into different Resource Groups. * Benefit: You can set a

cpu_core_limit

or

mem_limit

for the heavy workload group. This prevents a single heavy query from saturating all nodes and allows the autoscaler to react to the "capped" load more predictably. * Hard Limits: Since v3.3.5+, StarRocks supports hard CPU limits (

exclusive_cpu_cores

), which can prevent a heavy query from "dragging down" the rest of the cluster performance while the autoscaler works. C. Adjusting HPA "Scaling Up" Sensitivity If the workload is bursty but not strictly scheduled, you can tune the Kubernetes HPA (Horizontal Pod Autoscaler) behavior: * Metric: Consider scaling on a 90th percentile CPU metric (using a Prometheus Adapter) rather than a simple average. This prevents idle nodes from masking the saturation on the "working" nodes. * Stabilization Window: Shorten the

scaleUp.stabilizationWindowSeconds

to react faster, though this increases the risk of flapping. D. Pre-emptive Warming (The "Pulse" Strategy) Some users trigger a "dummy" light query that hits all nodes right after scaling to "warm" the data cache on new nodes. While this doesn't help the current query, it ensures the next query in the pipeline benefits immediately from the new capacity and populated local caches. Summary The behavior you're seeing is a known architectural limitation regarding plan fixity. Until StarRocks introduces "Dynamic Plan Rescheduling" (a complex feature), the best approach is to decouple the scaling trigger from the execution of the first saturating query: 1. Use Scheduled Scaling for known windows. 2. Enable Query Queuing to ensure that once nodes are added, the next queries in line can actually use them rather than everyone fighting for the original 1-2 saturated nodes. Your upstream issue (#76485) is the right place to track the long-term fix for dynamic re-distribution. References • quick_start/shared-data.md • faq/shared_data_faq.md • deployment/shared_data/feature-support-shared-data.md

@Rocky is there a document or efforts planned for "Dynamic Plan Rescheduling" support?

The short answer is: there is no public design document for "Dynamic Plan Rescheduling" yet. However, your issue (GitHub #76485) highlights a core architectural challenge that the team is aware of. Here is the current landscape of efforts in this direction: 1. Related Roadmap Efforts While "mid-query node addition" isn't a single feature yet, the 2025/2026 Roadmap (Issue #55526) includes several "Dynamic" execution goals that overlap with this problem: * Dynamic Data Skew Handling: Developing algorithms to detect and mitigate skew during execution. This is the first step toward moving work between instances. * Adaptive Query Execution (AQE): StarRocks has begun introducing AQE (e.g.,

cbo_enable_dp_join_reorder

in v3.2), which allows the engine to change the plan (like join types or orders) based on actual row counts observed at runtime. * Refinement of Pipeline Engine: Ongoing work in the pipeline engine aims to make fragment execution more "preemptible," which is a prerequisite for moving work to new nodes. 2. The Current Technical Reality In StarRocks' current MPP architecture: * Planning is Static: The FE determines the number of fragment instances based on the backends available at the moment the query starts. * No Mid-Query Re-sharding: Once a shuffle (Exchange) is established between nodes $A$ and $B$, adding node $C$ mid-stream would require $A$ and $B$ to pause, recalculate hash buckets, and redirect part of their stream to $C$. This is a massive shift from the current high-performance "push" model. 3. Suggested Next Steps Since you have already filed the upstream issue, the developer discussion there will be the primary "document" as it evolves into an RFC (Request for Comments). What to watch for: * RFCs in the StarRocks repo: Keep an eye out for any design proposals labeled

type/RFC

or

area/optimizer/execution

. * Community Dev Meetings: This topic is a frequent candidate for the monthly community developer syncs. For now, the Scheduled Scaling or Query Queuing (to ensure new nodes are "caught" by the next batch of queries) remain the recommended production workarounds. References • best_practices/query_tuning/query_planning.md • sql-reference/System_variable.md • administration/management/monitoring/metrics.md • best_practices/query_tuning/query_profile_operator_metrics.md • using_starrocks/caching/query_cache.md