http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> what are valid reasons to make enab...
# questions-and-troubleshooting
p
@Rocky what are valid reasons to make enable_runtime_adaptive_dop true?
r
The session variable 
enable_runtime_adaptive_dop
(introduced in StarRocks v3.2.0) is used to allow the execution engine to dynamically adjust the Degree of Parallelism (DOP) of query fragments while they are running, rather than relying solely on static planning. Here are the valid reasons and scenarios to set `enable_runtime_adaptive_dop = true`: 1. Resource Efficiency in High-Concurrency Environments In clusters with many concurrent users, setting a high static 
pipeline_dop
can lead to CPU thrashing and resource exhaustion. When 
enable_runtime_adaptive_dop
is enabled, StarRocks can scale down the parallelism of specific fragments if the system is under heavy load or if the fragment's workload doesn't justify high resource consumption. This helps maintain high throughput for the entire cluster. 2. Handling Inaccurate Optimizer Estimates The StarRocks Cost-Based Optimizer (CBO) estimates the amount of data a query will process to decide the initial parallelism. However, if statistics are stale or the query is extremely complex, the estimate might be significantly off. * Small Data, Large Plan: If the CBO predicts a large amount of data but the actual rows are few, adaptive DOP can reduce parallelism at runtime to save CPU and overhead. * Large Data, Small Plan: Conversely, it can help prevent a query from being "under-parallelized" when it encounters more data than expected. 3. Dynamic Join Optimization When this feature is enabled, StarRocks can adapt the probe side of a join based on the actual size of the build side (hash table) completed at runtime. * Wait for Build: It forces join probes to wait for all build phases to complete. This allows the system to know exactly how much data is in the hash table before deciding how many threads the probe side should use. 4. Simplifying Manual Tuning Instead of DBAs having to manually tune 
pipeline_dop
(e.g., setting it to 1 for small queries and 16 for large ones), the system attempts to find the "sweet spot" automatically. This is particularly useful for mixed workloads where short, interactive queries and long, heavy ETL-style queries run on the same cluster. ────────── Important Trade-offs & Considerations Before enabling it, you should be aware of its side effects: * Disables Certain Filters: Enabling this currently disables pipeline-level multi-partitioned runtime filters, which might actually slow down certain join-heavy queries that rely on those filters. * Join Probe Latency: Since join probes must wait for the build phase to finish entirely, you lose the "streaming" benefit where probing starts as soon as the first part of a hash table is built. * Requires Pipeline Engine: This variable only works when 
enable_pipeline_engine
is set to 
true
. * Ignored if pipeline_dop > 0: If you have manually set 
pipeline_dop
to a specific number (like 
SET pipeline_dop = 8
), this adaptive setting is ignored. It only works when 
pipeline_dop
is 
0
(the default). Recommendation: Use it in environments with highly unpredictable query patterns or extremely high concurrency where resource contention is a frequent issue. If your workload consists of very stable, well-understood queries, you may get better performance by leaving it 
false
and letting the standard 
pipeline_dop=0
logic handle it. References • sql-reference/System_variable.md • knowledge/usage/best-practice.md
p
@Rocky So use this only when 
pipeline_dop
and 
max_pipeline_dop
is used?
r
Actually, it is exactly the opposite: You should use 
enable_runtime_adaptive_dop
only when you do not manually set 
pipeline_dop
to a fixed number. Here is how these three variables interact: 1. The Pre-condition: 
pipeline_dop
must be 0 If you set 
pipeline_dop
to any value greater than 0 (e.g., 
SET pipeline_dop = 8
), StarRocks treats this as a hard instruction to use exactly 8 threads. In this case, 
enable_runtime_adaptive_dop
is ignored. * Recommendation: Keep 
pipeline_dop = 0
(the default) to allow the system to choose the parallelism. 2. The Ceiling: 
max_pipeline_dop
 When 
pipeline_dop
is 0, StarRocks calculates the parallelism automatically. On machines with very high core counts (e.g., 128 cores), the auto-calculation might set the DOP too high (e.g., 64), which can cause overhead. * 
max_pipeline_dop
acts as a "safety cap" or ceiling. It ensures that the automatically calculated DOP never exceeds this value. * Even with 
enable_runtime_adaptive_dop = true
, the runtime adjustment will still respect the 
max_pipeline_dop
limit. 3. The Logic: 
enable_runtime_adaptive_dop
 While 
max_pipeline_dop
provides a static upper limit, 
enable_runtime_adaptive_dop
provides runtime flexibility. * Without this enabled, the DOP is fixed at the start of the query based on the optimizer's estimates. * With this enabled, the system can look at the actual data size (e.g., after a Join Build phase is finished) and decide to reduce or increase the DOP for the subsequent phases (like the Join Probe) to be more efficient. Summary of when to use them together: Variable | Value | Purpose 
pipeline_dop
| 
0
| Mandatory. Allows the system to manage parallelism automatically. 
max_pipeline_dop
| 
64
(default) | Optional. Adjust this if you want to lower the "ceiling" for all queries on a high-core machine. 
enable_runtime_adaptive_dop
| 
true
| Recommended. Enables the system to refine that parallelism while the query is actually running. Bottom line: You use 
enable_runtime_adaptive_dop
to make the "Automatic" mode (
pipeline_dop = 0
) smarter. You don't use it if you are manually pinning your queries to a specific parallelism. References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • faq/resource_isolation_faq.md
p
@Rocky when 
pipeline_dop=0
how does it figure out what dop to set for each query?
r
When 
pipeline_dop
is set to 
0
(the default), StarRocks switches from a "fixed" parallelism to an automatic/adaptive parallelism mode. In this mode, the system calculates the Degree of Parallelism (DOP) for each query fragment using several factors: 1. The "Base" Calculation (Hardware-Aware) By default, the system calculates a baseline DOP based on your hardware. * The Rule of Thumb: It typically defaults to half the number of physical CPU cores on the BE (Backend) nodes. * The Ceiling: This value is always capped by the session variable 
max_pipeline_dop
(which defaults to 64). Even if you have 256 cores, the auto-DOP will not exceed 64 unless you increase this limit. 2. The Cost-Based Adjustment (Query-Aware) The StarRocks Cost-Based Optimizer (CBO) analyzes the query plan and estimates the "cost" (CPU, memory, and data volume) of each fragment. * Small Queries: If the optimizer estimates that a query will process a very small number of rows (e.g., a simple lookup or a small scan), it will automatically set a lower DOP (often 1 or 2). This prevents the "over-parallelization" overhead where the time spent managing threads exceeds the time spent processing data. * Large Queries: For heavy scans or complex joins, it scales the DOP up toward the baseline/ceiling. 3. Data Distribution (Tablet/Morsel Count) The DOP is also physically limited by how the data is stored: * Tablet Count: If a query fragment is scanning a table with only 4 tablets, it cannot effectively use a DOP of 64 for the scan operator. The system will adjust the DOP to match the available data parallelism (the number of "morsels" or units of work). 4. Dynamic Adjustment (v3.1.4+ and v3.2+) In newer versions, StarRocks has become even more "elastic" regarding DOP: * Query Queue Integration (v3.1.4+): If you have Query Queues enabled, the FE (Frontend) monitors how many queries are currently running. If the system is busy, it will dynamically lower the DOP for new incoming queries to ensure the BEs aren't overwhelmed by too many total "drivers" (threads). * Runtime Adaptive DOP (v3.2+): As discussed earlier, if 
enable_runtime_adaptive_dop
is 
true
, the engine can even change the DOP during execution. For example, it might start a join probe with a low DOP if the join build phase resulted in a very small hash table, regardless of what the initial plan suggested. How to verify the DOP of a query? To see what DOP StarRocks actually chose for a specific query, you can look at the Query Profile: 1. Run 
set enable_profile = true;
2. Execute your query. 3. Run 
get_profile(query_id);
(or check the FE web UI). 4. Search for the 
DegreeOfParallelism
metric in the Pipeline sections. Summary table for 
pipeline_dop=0
 Factor | Influence Physical Cores | Sets the standard target (usually 
cores / 2
). 
max_pipeline_dop
 | Hard upper limit (default 64). Estimated Cost | Lowers DOP for "light" queries to reduce overhead. Data Locality | Adjusts DOP based on the number of Tablets/Morsels. Concurrency | Lowers DOP when many queries are already running (if Query Queues are on). References • best_practices/query_tuning/query_planning.md • best_practices/query_tuning/query_profile_operator_metrics.md • using_starrocks/caching/query_cache.md • using_starrocks/Cost_based_optimizer.md
p
@Rocky what is lowest dop it will ever set to running query?
r
In StarRocks, the lowest Degree of Parallelism (DOP) that the system will ever set for a running query (or a specific fragment within that query) is 1. Here is the breakdown of how that "floor" is reached: 1. The Absolute Minimum: DOP 1 The Degree of Parallelism represents the number of execution threads (drivers) for a query fragment. Since a query must run at least one thread to process data, the DOP cannot drop below 1. * When it happens: StarRocks will set the DOP to 1 if the Cost-Based Optimizer (CBO) determines the query is "small" (e.g., a simple point lookup or a scan with very few rows) or if you manually set 
SET pipeline_dop = 1;
. 2. The "Small Query" Threshold When 
pipeline_dop
is set to 
0
(automatic), the system doesn't just jump from 0 to 64. For many simple queries, it will default to a DOP of 1 or 2. This is a deliberate optimization to avoid the "scheduling overhead" of managing 64 threads when the work could be finished by a single thread in milliseconds. 3. Query Queue Driver Floor (Aggregate Parallelism) While a single fragment can have a DOP of 1, StarRocks' Query Queue (v3.1.4+) introduces a concept of a "driver floor" for the entire query across a Backend (BE) node. * If your system is under heavy load and the query queue is managing concurrency, it will try to scale down new queries to their minimum resource usage. * The system uses internal limits (like 
query_queue_cpu_used_permits_min_limit
) to ensure a query isn't squeezed so small that it becomes inefficient. Even under extreme congestion, it will still allow the query to run with at least the minimum number of threads required to make progress (the "drivers"). Summary * Lowest possible per-fragment DOP: 1 * Lowest auto-calculated DOP for "Light" queries: 1 or 2 * Why not 0? A DOP of 0 would mean the query is paused or has no threads assigned to it, which only happens if the query is sitting in a Pending Queue waiting for resources to become available. Once it starts "running," the DOP is at least 1. Rocky's Tip: If you see your queries running with a DOP of 1 in the profile and you think they should be faster, it's usually because the optimizer thinks the table is smaller than it actually is. In that case, running 
ANALYZE TABLE <table_name>
to update statistics can help the system "authorize" a higher DOP. References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • faq/resource_isolation_faq.md • administration/management/FE_configuration.md
p
@Rocky what does increasing 
query_queue_v2_concurrency_level
 end up doing in shared-data mode?
r
In StarRocks, the parameter 
query_queue_v2_concurrency_level
is a key tuning knob for the Query Queue V2 (the newer, slot-based scheduler). When you increase this value in shared-data (lake) mode, several specific things happen to your resource management and throughput: 1. It Expands the Cluster's "Total Slots" The parameter essentially tells the Frontend (FE) how many parallel "layers" of queries it should allow relative to your CPU cores. In shared-data mode, StarRocks uses this to calculate the total number of concurrent query slots available. * The Shared-Data Multiplier: In shared-data mode (specifically when multi-warehouse is active), the effective concurrency level is scaled down by a factor of 4: $$\text{Effective Level} = \max(1, \frac{\text{query\_queue\_v2\_concurrency\_level}}{4})$$ * End Result: If you increase this value, the FE will allow more queries to start executing simultaneously across your compute nodes (warehouses) before it begins placing new queries into the queue. 2. It Shrinks the "Memory Budget" per Query The Query Queue V2 uses a "slot-based" accounting system. Every query is assumed to take up a certain amount of memory based on the total available memory divided by the number of slots. * The Math: $\text{Memory per Slot} = \frac{\text{Worker Memory}}{\text{Cores} \times \text{Concurrency Level}}$ * End Result: Increasing the concurrency level makes the FE believe each query is "smaller" and cheaper. While this allows more queries to run, it also means the system is less conservative. If your queries are actually memory-intensive, increasing this too high can lead to Backend (BE) OOM (Out of Memory) crashes because the FE over-committed the available RAM. 3. It Impacts CPU Cost Accounting The V2 queue also tracks CPU usage using "permits." By increasing the concurrency level, you are essentially lowering the bar for how much "CPU weight" a single query carries in the scheduler's eyes. * End Result: More queries will be allowed to compete for CPU cycles. In shared-data mode, where compute nodes are often scaled up/down, this allows you to saturate the CPU of your warehouses more aggressively. 4. Behavioral Difference: Shared-Data vs. Shared-Nothing In a standard Shared-Nothing cluster, the default 
concurrency_level
of 
4
is usually sufficient. However, in Shared-Data mode: * Because of the 
/ 4
scaling factor mentioned in the documentation, a default value of 
4
results in an effective concurrency level of 
1
. * Why? This is often done because shared-data clusters are designed to be elastic. StarRocks defaults to a more conservative concurrency to ensure stability when warehouses are joining or leaving the cluster. * Recommendation: If you find that your shared-data cluster is queueing queries even when CPU and Memory usage are low, you likely need to increase 
query_queue_v2_concurrency_level
to 
8
or 
16
to compensate for the shared-data scaling factor. Summary Table: Increasing the Level Metric | Effect | Impact on Shared-Data Mode Concurrent Queries | Increases | More queries run in parallel per Warehouse. Queuing Frequency | Decreases | Queries spend less time in 
Pending
state. Memory per Query | Decreases | Risk: Higher chance of OOM if queries are heavy. CPU Saturation | Increases | Better utilization of high-core compute nodes. Rocky’s Tip: If you increase this value, keep a close eye on your BE memory usage (
SHOW PROC '/backends'
). If you see "Memory Exceed Limit" errors in your query results, you have likely set the concurrency level too high for your workload's query complexity. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md
p
@Rocky Why would lower tablet_sched_slot_num_per_path value end up with better latency during CN scales.
r
In StarRocks, lowering the 
tablet_sched_slot_num_per_path
value during Compute Node (CN) scaling helps maintain better query latency by throttling the speed of background rebalancing and reducing resource contention. While it might seem counterintuitive to slow down the scaling process, here is why a lower value improves the experience for active users: 1. Reducing RPC and Control Plane Pressure When a CN cluster scales (especially when adding nodes), the Frontend (FE) needs to redistribute tablet "assignments" to ensure the new nodes take their share of the work. * The Slot Limit: 
tablet_sched_slot_num_per_path
controls how many of these tablet-related tasks (Add/Drop/Move) can happen simultaneously per storage path. * The Impact: If this is set too high (e.g., the default of 8 or higher), the FE may flood the CNs with thousands of concurrent RPC requests to update tablet metadata. This creates "RPC noise" and high CPU overhead on the CN’s management threads, competing directly with the threads trying to execute your SQL queries. 2. Preventing "Cache Storms" (Specific to Shared-Data) In shared-data (Lake) mode, data is stored in object storage, but CNs maintain a Local Data Cache. * Tablet Migration = Cache Loss: When a tablet is "moved" from one CN to another to balance the load, the new CN starts with an empty cache for that tablet. * The Latency Spike: If you allow too many tablets to migrate at once (high slot count), a large portion of your "hot" data suddenly becomes "cold" across the cluster. This triggers a massive surge in remote I/O requests to S3/HDFS. * The Benefit of Lowering: By lowering the slots, you force the rebalancing to happen gradually. The cache misses are spread out over a longer period, preventing a sudden "latency wall" where every query is stuck waiting for remote object storage. 3. Mitigating Disk and Network Contention Even in shared-data mode, CNs use their local disks for metadata and caching. * I/O Fluctuations: Background tasks for tablet scheduling involve writing metadata files and managing cache segments. High concurrency here causes I/O fluctuations. * Bandwidth: Scaling creates a burst of network traffic for metadata synchronization. Throttling this ensures that the majority of your network bandwidth remains available for query results and data shuffling. Recommended Tuning If you prioritize stability and latency over "speed of scaling," you can temporarily lower this setting during a scaling event:
Copy code
sql
-- Lowering the concurrency to reduce impact on queries
ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "2");

-- You may also want to limit the total global concurrent tablets
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "100");
Summary: Lowering this value acts as a "speed limit" for the cluster's internal reorganization. It ensures that the background "heavy lifting" of moving tablets doesn't steal the CPU, Network, or Cache resources needed to keep your user queries fast. References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • administration/management/resource_management/Replica.md • administration/management/BE_configuration.md
p
@Rocky during a CN rebalancing does less work get from FE to CN for 15-30 seconds?
r
Yes, it is common to observe a 15–30 second "dip" or "stutter" in work dispatched from the FE to CNs during a rebalancing or scaling event. This isn't usually a total stop, but rather a bottleneck in the FE's Query Planning and Scheduling layer. Here is the technical breakdown of why this happens and why it specifically lasts for that 15–30 second window: 1. Metadata Lock Contention (The "Catalog Lock") The Frontend (FE) maintains a global metadata map (the 
TabletInvertedIndex
) that tracks exactly which tablets live on which CN nodes. * The Action: When you scale or rebalance, the 
TabletScheduler
must move tablets. To do this, it frequently acquires a Database Write Lock to update the catalog. * The Dip: Query Planning (the process of FE sending work to CNs) requires a Database Read Lock. If the rebalancing process is aggressive, the Write Lock requests from the scheduler can "queue up" behind active queries or block new ones, causing the FE to temporarily stop generating and dispatching new query fragments. 2. The 30-Second Transaction/Cleanup Cycle Several internal StarRocks "daemon" tasks are hardcoded or defaulted to 30-second intervals. * 
transaction_clean_interval_second
(Default: 30s): During rebalancing, hundreds of "migration transactions" are created. The FE performs a cleanup of these transaction states every 30 seconds. * 
table_keeper_interval_second
(Default: 30s): This daemon ensures partition properties and replication states are consistent. During a rebalance, it becomes much more active. * The Result: You see a 30-second "wave" where the FE is busy with background administrative overhead rather than foreground query dispatch. 3. The "Report & Heartbeat" Synchronization When a CN joins or tablets move, there is a mandatory "handshake" period: * Heartbeat (5s): The FE must confirm the CN is alive. * Task Report (10s): The CN reports the success of tablet migration tasks back to the FE. * Tablet Report (60s default, but triggered on change): The CN sends a full report of its local tablets. Before these cycles complete and the metadata is "published" across the FE followers, the Optimizer may still be using a "stale" version of the cluster map, leading to a temporary pause in sending new work to the recently changed nodes. 4. Query Queue V2 Re-calculation If you are using Query Queues, the FE must re-calculate the "Total Slots" and "CPU Permits" available across the cluster every time the node count changes. * In shared-data mode, the system uses a specific scaling factor (dividing total concurrency by 4). During the 15–30 seconds while the cluster state is "unstable" (nodes joining/leaving), the FE often defaults to a more conservative slot calculation, which results in fewer queries being released from the 
PENDING
queue to the CNs. How to Mitigate the Dip If this dip is causing production issues, you can "smooth out" the rebalancing by making it less aggressive: 1. Lower the scheduling speed:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "2");
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "100");
This makes the rebalancing take longer, but reduces the frequency and duration of the metadata lock contention. 2. Ensure FE Resources: If the FE is hitting 100% CPU during that 30-second window, it is likely struggling with the metadata updates. Ensure your FE has at least 8-16 cores. Summary: The 15-30 second window is the time it takes for the FE to process a batch of metadata transactions, clean up the task states, and receive the "Task Success" reports from the CNs via the 10s/30s internal daemon loops. References • administration/management/FE_configuration.md
p
@Rocky if FEs queue less does this help CNs stay busy during scaling?
r
To answer your question directly: Yes, having the FEs "queue less" (by increasing concurrency) can help CNs stay busy during the 15–30 second scaling "dip," but it is a double-edged sword that can lead to cluster instability. Here is the breakdown of how this interaction works during a scaling or rebalancing event in shared-data mode. 1. The "Buffer" Effect When you increase 
query_queue_v2_concurrency_level
, you are effectively allowing more queries to move from the FE's internal queue to the "Running" state on the CNs. * During the 15–30s Dip: The FE often hits a bottleneck where it is busy processing tablet reports, updating metadata locks, or cleaning up transactions. During this window, the FE's ability to plan and dispatch new work is reduced. * Why Queuing Less Helps: If you have already dispatched a high number of queries to the CNs (because your queue limit was high), those queries will continue to execute on the CNs even while the FE is "stuttering." This acts as a work backlog that keeps the CN CPUs busy while the FE is temporarily unable to send new tasks. 2. The Risk: "Resource Contention" While keeping the CNs busy sounds good, "queuing less" during a scaling event creates a high risk of Backend OOM or CPU exhaustion. * Shared-Data Factor: As mentioned previously, StarRocks shared-data mode scales the concurrency level by 1/4 by default. This is a safety margin. * The Conflict: When a CN is rebalancing, it is already performing high-overhead background tasks: 1. Metadata RPCs: Handling thousands of tablet updates from the FE. 2. Cache Storms: Fetching remote data from object storage for newly assigned tablets. 3. Local I/O: Updating local cache metadata. * The Result: If the FE "queues less" and floods the CN with a massive number of concurrent user queries at the exact same time the CN is trying to rebalance, the CN may run out of memory (OOM) because the FE's "per-slot memory" calculation didn't account for the extra memory used by the background rebalancing and cache-filling processes. 3. Better Alternatives to "Queuing Less" If your goal is to prevent the 15–30 second dip and keep the CNs productive without crashing them, consider these "smoothing" techniques instead of just increasing global concurrency: Strategy | Why it's better than "Queuing Less" Lower 
tablet_sched_slot_num_per_path
 | Slows down rebalancing so it doesn't "choke" the FE with metadata updates, preventing the dispatch dip from happening in the first place. Increase FE CPU/Memory | The 15–30s dip is often a sign that the Leader FE is struggling with Java GC or Lock Contention. Giving the FE more cores helps it process reports faster. Use Resource Groups | Instead of increasing global concurrency, increase it only for specific "small query" groups. This keeps CNs busy with light work while heavy queries remain queued, protecting the system from OOM. Summary If you increase 
query_queue_v2_concurrency_level
, you will likely see higher CN utilization during the scaling dip because you've "pre-loaded" them with work. However, if that work is memory-intensive, you are significantly increasing the chance that your CNs will crash or experience massive latency spikes as they struggle to balance user SQL and background tablet migrations simultaneously. Rocky's Recommendation: If you see the 15–30s dip, first try to throttle the rebalancing speed (
tablet_sched_slot_num_per_path
= 2) rather than opening the query floodgates. It's better to have a slightly slower scaling process than a cluster that is "busy" but failing queries. References • administration/management/Scale_up_down.md • administration/management/resource_management/Replica.md
p
@Rocky I already have 
tablet_sched_slot_num_per_path
at 1.
r
Since you have already set 
tablet_sched_slot_num_per_path
to 1, you have essentially reached the "minimum speed" for per-disk scheduling. However, if you are still seeing that 15–30 second "dip" in query dispatching, it means the bottleneck has shifted from BE/CN resource contention to FE Metadata Lock Contention. Even with a value of 1, the cluster can still struggle if it is trying to do too many things globally. Here is why you still see the dip and what you can tune next: 1. The "Total Concurrent Tablets" Bottleneck While 
tablet_sched_slot_num_per_path
limits work per disk, the total number of tablets being rebalanced across your entire cluster can still be very high. * The Parameter: 
tablet_sched_max_balancing_tablets
(Default: 500). * The Issue: If you have 50 CNs, even with 1 slot per path, the FE might be trying to manage 500 active tablet migrations simultaneously. Each time one of these finishes, the FE must acquire a Write Lock on the database metadata to update the tablet's location. * The Fix: Lower this global limit.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "50");
This forces the FE to handle fewer metadata updates per second, leaving more "lock time" available for query planning and dispatching. 2. FE Edit Log (BDBJE) Write Latency Every time a tablet finishes moving, the FE leader must write an entry to its edit log (using BDBJE) and synchronize it to the follower FEs. * The 15–30s Window: If the FE is flooded with hundreds of "Migration Completed" reports at once, the edit log can become a bottleneck. While the FE is waiting for these disk I/O operations to sync the metadata changes, it often pauses query dispatching. * Observation: Check the FE 
fe.log
. If you see many 
slow edit log write
warnings or 
take XXX ms to commit
messages during scaling, your FE metadata disk (where the 
meta/
directory lives) might be too slow. 3. The 
ReportHandler
Lock Contention StarRocks has an internal thread called the 
ReportHandler
that processes heartbeats and tablet reports from CNs. * The Conflict: During scaling, CNs send frequent "Tablet Reports" to tell the FE "I now have these tablets." The 
ReportHandler
needs to hold a lock to process these. If the reports are massive (many tablets), it can block the 
QueryPlanner
from accessing the metadata it needs to send work to the CNs. * Why the dip happens at 15-30s: This is the typical interval for the FE to process a full "batch" of these reports and commit the state changes. 4. Shared-Data "Metadata-Only" Rebalancing If you are in Shared-Data (Lake) mode, rebalancing is "metadata-only." It doesn't move data, just the "ownership" of the tablet. * The Paradox: Because "moving" a tablet is so fast (just a metadata update), the FE can actually process them too quickly. Setting the slot to 1 helps, but if you have thousands of tablets, the FE still gets overwhelmed by the sheer volume of "Success" reports. * Solution: In addition to the settings above, you can temporarily disable balancing during critical high-traffic windows and re-enable it during off-peak hours:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_disable_balance" = "true");
Summary of Next Steps If 
slot_num_per_path = 1
isn't enough to stop the dip, use these "stronger" throttles: 1. Reduce Global Concurrency: Set 
tablet_sched_max_balancing_tablets
to a very low number (e.g., 20 or 50). 2. Check FE Hardware: Ensure the FE's metadata directory is on a high-performance SSD/NVMe. 3. Check FE CPU: If FE CPU spikes to 100% during the dip, it's likely JVM Garbage Collection or Lock Contention; increasing FE cores can help the FE "work through" the metadata update faster. 4. Monitor Locks: Run 
SHOW PROC '/com.starrocks.catalog.Database';
(if available in your version) or check 
fe.audit.log
for high 
QueryPlanTimeMs
.
By lowering the global balancing limit, you ensure the FE spends 95% of its time on user queries and only 5% on background rebalancing, which should smooth out that 30-second dispatching gap. References • knowledge/trouble_shooting/fe_deadlock_problems.md
p
@Rocky For our CN scaling out I was looking to try following "scale out when CPU > 50% plus either latency ratio > 1.75, low cache health, or rising publish-version backlog for 30 seconds". Which custom metrics for starrocks should I consider
r
For a StarRocks CN (Compute Node) scaling strategy, you should focus on metrics that distinguish between transient spikes and sustained resource exhaustion. Since you are using shared-data mode, the following Prometheus/StarRocks metrics are most relevant for your logic: 1. CPU Usage (> 50%) StarRocks does not always export a "total CPU %" metric directly as an internal gauge; it is often better to use standard Prometheus node-exporter metrics or the resource-group specific metrics: * 
starrocks_be_resource_group_cpu_use_ratio
: If you use resource groups, this tracks CPU usage per group. * System Metric: 
1 - avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m]))
is the industry standard for auto-scaling triggers. 2. Latency Ratio (> 1.75) To calculate a ratio, you need a baseline (e.g., P95 latency over 1 hour) compared to the current P95: * 
starrocks_fe_query_resource_group_latency
: (Labels: 
quantile="0.95"
, 
resource_group="xxx"
). * Alternative: 
starrocks_fe_query_latency_ms
(Summary metric). * Logic: 
(Rate of latency last 5m) / (Rate of latency last 1h) > 1.75
. 3. Cache Health (Low Health) In shared-data mode, "health" is usually a combination of hit rate and disk space. If the cache hit rate drops while CPU is high, it means the CN is spending too many cycles on remote I/O (S3/GCS/Azure) rather than compute. * 
block_cache_hit_rate_last_minute
: This is the most sensitive metric for scaling. A drop here indicates "cache thrashing" or a shift in the working set. * 
datacache_disk_used_bytes
vs 
datacache_disk_quota_bytes
: If 
used / quota
is near 1.0 and the hit rate is low, your current CN fleet's local disk cache is too small for the active dataset. 4. Publish-Version Backlog (30 Seconds) A "backlog" here can be measured by the number of tasks in the queue or the latency of the publish phase. If this rises, it means the CNs are too busy to acknowledge metadata updates from the FE. * 
publish_version_queue_count
: The number of version tasks queued on the CN. A sustained value > 0 for 30s is a strong "busy" signal. * 
starrocks_fe_txn_publish_latency_ms
: (Labels: 
quantile="0.95"
). This is reported by the FE. If the 95th percentile of publish latency climbs, it means the CNs are struggling to keep up with ingestion commits. Recommended Scaling Rule (PromQL style)
Copy code
promql
# Primary Trigger: High CPU
(avg(node_cpu_usage_percent) > 50) 
AND (
  # Choice A: Latency Spike
  (label_replace(starrocks_fe_query_latency_ms{quantile="0.95"}, "type", "current", "", "") / 
   avg_over_time(starrocks_fe_query_latency_ms{quantile="0.95"}[1h])) > 1.75
  OR
  # Choice B: Cache is failing
  (block_cache_hit_rate_last_minute < 0.6)
  OR
  # Choice C: Publish Backlog
  (publish_version_queue_count > 10)
)
Pro-Tip: The "Cool-down" Since you mentioned the 15–30 second dip earlier, ensure your scaling policy has a stabilization window (e.g., "for 2 minutes"). You don't want the 30-second rebalancing "stutter" to trigger a second, unnecessary scale-out event while the first one is still finalizing its metadata. References • administration/management/monitoring/metrics.md • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md References • using_starrocks/caching/block_cache.md • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md
p
@Rocky — We're designing CN scale-out for a shared-data StarRocks cluster on Kubernetes (StarRocks operator + HPA, Query Queue v2 enabled, 
run_mode=shared_data
). Today our HPA mostly keys off `starrocks_fe_query_queue_pending`; in benchmarks, CPU-only scaling reacted too late (CN saturation, 
starlet ... Deadline Exceeded
before scale-up). We want a compound scale-out rule (sustained 30s):
Scale out 1 CN when average CN CPU > 50%
AND at least one of:
Query latency ratio > 1.75(current p95 vs recent baseline, or your recommended definition)
Low cache health(datacache / metacache / fslib cache under pressure)
Rising publish-version backlog(queue depth or schedule latency trending up)
Questions: 1. For each signal above, whichexact Prometheus metric namesshould we use (FE vs CN, labels, aggregation —`avg`across CNs,
max
, per-warehouse, etc.)? 2. Forlatency ratio, is there a native StarRocks metric, or should we derive it from something like`starrocks_fe_query_resource_group_latency`/ query latency histograms? What baseline window do you recommend? 3. Forcache healthon shared-data CNs, should we prefer: ◦ `datacache_mem_used_bytes`/`datacache_mem_quota_bytes` ◦ 
query_cache_hit_ratio
◦ Starlet`fslib cache hit ratio`/ cache misses ◦ Metacache miss rates (segment / metadata / txn log) — and what thresholds indicate “cache-starved, add CN” vs “normal”? 4. Forpublish-version backlog, are`publish_version_queue_count`,
starrocks_fe_txn_publish_schedule_latency_ms
, and/or shared-data dashboard “Queued Tasks” the right signals? Which best predicts query degradation under load? 5. Should we also consider newer queue/slot metrics like`query_queue_pending_sum_raw_slots`(per-warehouse) instead of or in addition to`starrocks_fe_query_queue_pending`? 6. Can thisAND + ORpolicy be expressed inK8s HPA v2directly, or do you recommendKEDA + PromQL recording rules/ external autoscaler? 7. Anyanti-patternswith Query Queue v2 + HPA (e.g. scaling on queue depth while`query_queue_v2_concurrency_level`is still absorbing load)? Cluster context: EKS, CN pods ~46 CPU / 80Gi, HPA min/max CNs (e.g. 4→12), CN registration delay entrypoint override, datacache enabled. Workload is interactive SQL (commission analytics), not heavy ingest.
r
This compound scaling logic is highly effective for shared-data clusters where CPU saturation is often a lagging indicator of performance degradation. 1. Prometheus Metrics Mapping For a 30s sustained trigger, use the following metrics (all should be aggregated with 
avg
across CNs unless specified): Signal | Prometheus Metric Name | Source | Aggregation Recommendation CN CPU | 
starrocks_be_cpu_usage
| CN | 
avg by (cluster_name)
Latency | 
starrocks_fe_query_latency_ms
| FE | 
rate(...)
or 
quantile=0.95
Cache Hit | 
starrocks_be_block_cache_hit_ratio
| CN | 
avg
(Shared-data unified cache) Publish Q | 
publish_version_queue_count
| CN | 
max
(One saturated CN can block a txn) Publish Latency | 
starrocks_fe_txn_publish_latency_ms
| FE | 
quantile=0.95
(Leader FE only) 2. Latency Ratio Calculation There is no "native" ratio metric; you should derive it in your PromQL. * Recommended Logic: 
(avg_over_time(starrocks_fe_query_latency_ms{quantile="0.95"}[5m])) / (avg_over_time(starrocks_fe_query_latency_ms{quantile="0.95"}[1h])) > 1.75
* Baseline Window: 1 hour is recommended for interactive SQL. It is long enough to ignore transient spikes but short enough to adapt to daily trend shifts. 3. Cache Health for Shared-Data In shared-data mode, "Data Cache" is the primary bottleneck. * Preferred Metric: 
starrocks_be_block_cache_hit_ratio
(or calculate via 
block_cache_hit_bytes
vs 
block_cache_miss_bytes
). * Threshold: For interactive analytics, a drop below 0.70 (70%) usually triggers a "cache-starved" state where queries wait on S3/EBS I/O. * Metacache: 
starrocks_be_metacache_usage_ratio
> 0.90 is a secondary signal, but usually, Data Cache thrashing hits performance much earlier. 4. Publish-Version Backlog * Primary Signal: 
publish_version_queue_count
(CN-side). If this is
10
 for 30s, CNs are too busy executing fragments to acknowledge metadata updates. * Best Predictor: 
starrocks_fe_txn_publish_schedule_latency_ms
. This measures the time a transaction sits in the FE waiting for CNs to be ready. If this trends up, your CNs are effectively "unresponsive" to the FE leader, which is the #1 predictor of imminent 
Deadline Exceeded
errors. 5. Queue v2 "Slot" Metrics Yes, prefer 
starrocks_fe_query_queue_pending_sum_raw_slots
. In StarRocks 3.x with Queue v2, a single "heavy" query might request 20 slots (vCPUs), while a light query requests 1. Scaling on 
query_queue_pending
(count) treats them as equal. * Why use it: Scaling based on 
sum_raw_slots
allows the HPA to understand the magnitude of the resource deficit. If you have 100 pending slots and each CN provides 46, you know exactly how many CNs to add. 6. K8s HPA vs. KEDA Recommendation: KEDA. K8s HPA v2 supports multiple metrics, but it uses MAX logic (if any metric is above target, scale). It does not natively support the 
(CPU > 50% AND (A OR B))
logic you described. * KEDA Implementation: Use a 
ScaledObject
with a 
prometheus
trigger. Put your entire compound logic into a single PromQL expression that returns a value > 1 when the condition is met. 7. Anti-patterns with Queue v2 * Scaling on "Absorbable" Load: Query Queue v2 is designed to keep the system stable during 5-10 second bursts. Setting HPA to react too fast (e.g., 
< 15s
) will cause "flapping" where you scale up for a burst that would have cleared naturally. * The "Rebalance Dip": When a new CN joins, the FE briefly pauses dispatching to update tablet locations. If your HPA is keyed purely on 
pending_queries
, it might see a spike during this 15s window and try to scale again, leading to over-provisioning. * Fix: Use a 
stabilizationWindowSeconds
of at least 300s (5 mins) for scale-down and 60s for scale-up to allow the metadata rebalance to settle. Summary PromQL for KEDA: ```promql # Returns 1 if scaling is needed, 0 otherwise (avg(starrocks_be_cpu_usage) > 50)
AND ( (avg_over_time(starrocks_fe_query_latency_ms{quantile="0.95"}[5m]) / ignoring(quantile) avg_over_time(starrocks_fe_query_latency_ms{quantile="0.95"}[1h]) > 1.75) OR (avg(starrocks_be_block_cache_hit_ratio) < 0.7) OR (max(publish_version_queue_count) > 10) ) ``` References • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md • administration/management/monitoring/metrics-shared-data.md • deployment/shared_data/feature-support-shared-data.md
p
@Rocky you told me to look at starrocks_fe_query_latency_ms but can't find where that is documented.
r
You are correct to be cautious—the name 
starrocks_fe_query_latency_ms
is often used in StarRocks Alerting Guides and community Grafana dashboards, but it may not appear in the top-level "Metric Items" table in some versions of the official documentation because it is part of the Resource Group or Summary metric sets. In modern StarRocks (v3.x+), the documentation emphasizes granular metrics. Here is the exact breakdown of where to find this and what to use: 1. Where it is documented You can find the technical definition in the official docs under General Monitoring Metrics. Specifically, look for 
starrocks_fe_query_resource_group_latency
. * Why the name differs: In StarRocks, almost all queries are now tracked within "Resource Groups" (even if you only have the 
default_wg
). * The Metric: 
starrocks_fe_query_resource_group_latency
* Labels: 
quantile
(e.g., 
0.95
, 
0.99
), 
resource_group
. * Unit: Milliseconds (ms). 2. Why you might see 
starrocks_fe_query_latency_ms
 If you are using an older version or certain community-contributed Prometheus exporters, the global query latency is sometimes exported without the resource group dimension as a legacy metric. However, for your HPA/KEDA scaling logic, you should use the Resource Group version as it is the most stable and modern implementation:
Copy code
promql
# To get the P95 latency for your cluster:
starrocks_fe_query_resource_group_latency{quantile="0.95", resource_group="default_wg"}
3. Summary vs. Gauge Note that this is a Summary type metric. When you query Prometheus, you won't just see one number; you will see: * `starrocks_fe_query_resource_group_latency{quantile="0.95"}`: The 95th percentile value. * `starrocks_fe_query_resource_group_latency_sum`: Total time of all queries (useful for calculating average). * `starrocks_fe_query_resource_group_latency_count`: Total number of queries. Recommendation for your Scaling Logic Since your goal is to detect relative degradation (the "Ratio > 1.75"), use this PromQL which targets the 95th percentile specifically:
Copy code
promql
# Current 5-minute P95 latency / 1-hour P95 baseline
(avg_over_time(starrocks_fe_query_resource_group_latency{quantile="0.95"}[5m]))
/ 
(avg_over_time(starrocks_fe_query_resource_group_latency{quantile="0.95"}[1h]))
One final tip: If you don't see 
starrocks_fe_query_resource_group_latency
in your Prometheus target, check your 
fe/conf/fe.conf
. Ensure 
enable_resource_group = true
(default in 3.x) is set, as this is the engine that generates these specific latency metrics. References • administration/management/monitoring/metrics.md
2 Views
Open in Slack
PreviousNext
Powered by