http://starrocks.io logo
Join Slack
Powered by
hi <@U0A71G31CDV> When pipeline_dop is set to 0 (...
# questions-and-troubleshooting
r
hi @Rocky When pipeline_dop is set to 0 (auto), StarRocks 4.0.4 documentation says it becomes "half of the CPU cores". On K8s where CN runs inside a Pod with cgroup CPU limit = 8 (but host node has 64 physical cores), does the Pipeline Engine read the cgroup limit or the host core count? Which value decides the auto DOP?
r
In StarRocks 4.0.4, the Pipeline Engine is cgroup-aware. When running in a containerized environment like Kubernetes, it detects and respects the CPU limits (quota/period) defined in the Pod's resource configuration rather than the physical host's core count. 1. Detection Mechanism When the StarRocks Backend (BE) or Compute Node (CN) starts, it initializes its internal 
CpuInfo
by checking the cgroup control files (e.g., 
/sys/fs/cgroup/cpu/cpu.cfs_quota_us
and 
cpu.cfs_period_us
for cgroup v1, or 
cpu.max
for cgroup v2). * Host Core Count (64): Ignored for the purpose of capacity planning if a limit is set. * Cgroup Limit (8): Used as the basis for the engine's "perceived" core count. You can verify the number of cores StarRocks has detected by running:
Copy code
sql
SHOW BACKENDS; -- or SHOW COMPUTE NODES;
Look for the 
CpuCores
 column. In your scenario, it should display 8. 2. Auto DOP Calculation When 
pipeline_dop
is set to 
0
(auto), the effective Degree of Parallelism (DOP) is calculated using the following logic: * Formula: 
Effective DOP = min(max_pipeline_dop, detected_cpu_cores / 2)
* Calculated Value: 
min(64, 8 / 2) = 4
In your specific case, the auto DOP will be 4. 3. Why this matters This cgroup-aware behavior prevents the "Host Core Overscheduling" problem. If StarRocks used the host's 64 cores, an auto DOP of 32 would create too many execution threads for an 8-core Pod. This would lead to: 1. Severe CPU Throttling by the Kubernetes CFS (Completely Fair Scheduler). 2. High Context Switching overhead, degrading query performance. 3. Increased query latency due to "noisy neighbor" effects on the same host. Summary * Value used: The cgroup limit (8) decides the auto DOP. * Resulting DOP: 4 (half of the 8 cores allocated to the Pod). * Configuration Note: If you want to utilize all 8 cores more aggressively for single queries, you can manually set 
SET GLOBAL pipeline_dop = 8;
, but the "auto" setting (0) will stick to the safer "half-of-cores" rule to allow for background resources for other tasks like data loading and compaction. References • sql-reference/System_variable.md • faq/resource_isolation_faq.md • knowledge/usage/best-practice.md • administration/management/monitoring/metrics.md • deployment/sr_operator.md • best_practices/audit_log_resource_group.md
r
@Rocky so than, If a single query produces N fragments (e.g., 3-stage hash join = multiple fragments), does each fragment independently spawn pipeline_dop drivers on every BE/CN? So the total concurrent thread count for that query is approximately pipeline_dop * N * num_be_cn ? Or is there a cluster-wide cap?
r
To answer your second question: Yes, your logic is fundamentally correct. In the StarRocks Pipeline Engine, the degree of parallelism scales linearly with both the number of fragments and the number of nodes. Here is the breakdown of how these components multiply and where the limits reside: 1. The Multiplier Formula For a single query, the total number of Pipeline Drivers (the unit of execution in the pipeline engine) across the cluster is approximately: $$\text{Total Drivers} = \text{pipeline\_dop} \times \text{N (Logical Fragments)} \times \text{Number of BE/CN Nodes}$$ * Per BE/CN Node: On each node, for a single query, the number of drivers is 
pipeline_dop * (Number of fragments scheduled on this node)
. * Logical Fragments: A complex query (like a multi-stage join) is split into multiple "Fragments". Each fragment is then executed as one or more "FragmentInstances" on the BEs. * Drivers per Instance: Each FragmentInstance contains one or more Pipelines. The 
pipeline_dop
defines how many parallel Drivers are created for each of those pipelines to process data in parallel. 2. Is there a Cap? There isn't a single "cluster-wide hard cap" for a single query, but there are several safety mechanisms and thresholds: * The "Auto" Cap (
max_pipeline_dop
): Even if your BE has 128 cores, if 
pipeline_dop
is set to 
0
(auto), it is capped by the session variable 
max_pipeline_dop
(default is 64). This prevents a single query from spawning an astronomical number of drivers on high-core-count machines. * Per-BE Driver Queuing (
query_queue_driver_high_water
): Starting from v3.1.4, StarRocks uses "Query Queues" to manage driver saturation. * If the total number of running drivers on a BE exceeds the High Water Mark (default is 
avg_be_cpu_cores * 16
), the BE will start queuing new queries or dynamically lowering the effective DOP for incoming queries to prevent system exhaustion. * Thread Pool Limit: The Pipeline Engine uses a fixed-size thread pool (usually related to the number of CPU cores). While you can spawn 1,000 drivers, they will be multiplexed/scheduled onto a smaller number of physical worker threads. This prevents the "thread explosion" seen in older legacy engines. 3. Practical Example (Your K8s Pod) With your 8-core limit and 
pipeline_dop=0
(auto): * Effective DOP: 4 * Query with 3 Fragments: * Drivers per BE: $4 \text{ (DOP)} \times 3 \text{ (Fragments)} = 12$ drivers. * If you have 10 CNs: $12 \times 10 = 120$ drivers cluster-wide. Summary of Behavior Level | Multiplier | Controlled By Node Level | 
pipeline_dop
| 
pipeline_dop
/ 
max_pipeline_dop
Query Level | 
DOP * Fragments * Nodes
| N/A (Linear scaling) Safety Guard | BE-wide Driver limit | 
query_queue_driver_high_water
Rocky's Tip: If you notice high CPU utilization or "Slow Query" logs showing long 
ScheduleTime
, it often means the number of Fragments $\times$ DOP is too high for the available physical cores, causing drivers to wait in the ready queue for a free execution thread. In such cases, reducing 
pipeline_dop
to 
1
or 
2
for high-concurrency small-query workloads is a common practice. References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • administration/management/resource_management/query_queues.md • faq/Sql_faq.md • faq/resource_isolation_faq.md • administration/management/monitoring/alert.md
r
@Rocky In a Resource Group with concurrency_limit=4 and pipeline_dop=8, what is the actual concurrent driver count on a single CN? Is it 4 queries × 8 drivers = 32, or does concurrency_limit already account for pipeline_dop internally?
r
In StarRocks, the 
concurrency_limit
and 
pipeline_dop
variables operate at different layers of the execution engine. They do not account for each other internally in a way that limits the total "driver" count under a single parameter. 1. The Interaction The 
concurrency_limit
in a Resource Group defines the maximum number of active query instances allowed to run on a single CN/BE node for that group. If you have 
concurrency_limit=4
and 
pipeline_dop=8
, the actual concurrent driver count on a single CN is: $$\text{Actual Drivers} = \text{Running Queries} \times \text{pipeline\_dop} \times \text{N (Fragments on that node)}$$ In your example: * Queries: Up to 4 queries can run at once. * Drivers per Query Fragment: Each fragment of those queries will spawn 8 drivers. * Total Drivers: If each query has only 1 fragment running on that node, you will have 32 drivers. If each query has 3 fragments running on that node (e.g., for a complex join), you could have $4 \times 8 \times 3 = \mathbf{96}$ drivers. 2. Does StarRocks have a "Driver-Aware" Limit? Yes, but it is managed by the Query Queue system (available since v3.1.4), not the 
concurrency_limit
parameter itself. * `query_queue_driver_high_water`: This is a global or group-level "safety valve." If the total number of drivers across all running queries on a BE exceeds this threshold (default is 
avg_be_cpu_cores * 16
), StarRocks will: 1. Queue new incoming queries. 2. Dynamically reduce the 
pipeline_dop
of new queries to a value lower than your set 8 to prevent the system from being overwhelmed. 3. Summary for your Configuration * 
concurrency_limit
(4): Strictly limits the number of queries. If a 5th query arrives, it is either queued or rejected (depending on your 
enable_group_level_query_queue
setting). * 
pipeline_dop
(8): Strictly sets the parallelism for those 4 queries. * Result: You will have at least 32 concurrent drivers (4 queries × 8 DOP) active on the node. The system does not "auto-shrink" the query count just because the DOP is high, nor does it "auto-shrink" the DOP just because the query count is 4 (unless the High Water Mark mentioned above is hit). Rocky's Recommendation: On an 8-core Pod, having 32+ concurrent drivers is quite aggressive (a 4:1 ratio of drivers to physical cores). If these are CPU-intensive queries, you will likely see high context switching and CPU throttling. * If your queries are heavy, consider lowering 
pipeline_dop
to 4 or 
concurrency_limit
to 2. * If your queries are I/O bound (waiting on S3/HDFS), your current settings (32+ drivers) might be fine as many will be in a "blocked" state. References • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • best_practices/audit_log_resource_group.md
r
@Rocky When pipeline_dop is increased from 4 to 8, does exec_mem_limit apply per-driver (so total memory roughly doubles) or is exec_mem_limit a shared pool across all drivers of the same fragment? Please clarify the exact memory accounting model.
r
In StarRocks 4.0.4, the relationship between 
pipeline_dop
and memory limits is governed by a hierarchical memory tracking model. Here is the exact breakdown: 1. Does the limit double? No. The 
exec_mem_limit
(and its modern counterpart 
query_mem_limit
) is a shared budget, not a per-driver multiplier. * Shared Pool: If you set 
query_mem_limit = 4G
and increase 
pipeline_dop
from 4 to 8, the query still has exactly 4GB to use on that BE node. * Competition: The 8 drivers will now compete for that same 4GB pool. If the 8 drivers collectively exceed 4GB, the query will be killed with a "Memory Limit Exceeded" (MLE) error. 2. The Accounting Model: Hierarchical MemTrackers StarRocks manages memory using a tree-like structure of "MemTrackers." When a query runs on a CN/BE, the hierarchy looks like this: 1. Query Tracker (BE Level): Tracks the total memory used by all fragments of a single query on this specific node. This is where 
query_mem_limit
is enforced. 2. Fragment Instance Tracker: Tracks memory for one specific FragmentInstance. This is where 
exec_mem_limit
typically applies. 3. Operator/Driver Tracker: Tracks memory for individual operators (e.g., Hash Join, Aggregate) within a Pipeline Driver. Key Rule: When a driver allocates memory (e.g., for a hash table), it "charges" its parent Fragment Tracker, which in turn "charges" the Query Tracker. If any node in this hierarchy exceeds its limit, the allocation fails. 3. Impact of Increasing 
pipeline_dop
 While the limit stays the same, the actual memory usage almost always increases when you raise the DOP. This happens for three main reasons: * Concurrency Buffers: Each driver requires its own local exchange buffers and shuffle buffers. Doubling the drivers roughly doubles the memory used for data transit. * Data Structures: Some operators (like a Partitioned Hash Join) may create more small partitions or local states when DOP is higher, leading to higher metadata and pointer overhead. * Peak Convergence: With higher DOP, the query processes data faster. This can lead to a higher "Peak Memory" because more data is residing in memory simultaneously across more threads before it can be processed and released. 4. Summary Table: 
pipeline_dop
4 vs. 8 Feature | 
pipeline_dop = 4
 | 
pipeline_dop = 8
 
query_mem_limit
 | e.g., 4GB (Fixed) | 4GB (Fixed) Enforcement | Per Query on BE | Per Query on BE Actual Usage | Lower (e.g., 1.5GB) | Higher (e.g., 2.2GB) Efficiency | Slower, safer | Faster, higher risk of OOM Rocky's Advice: If you double your 
pipeline_dop
and start seeing "Memory Limit Exceeded" errors, it's not because the limit changed, but because the increased parallelism is consuming your "slack" memory more quickly. In this case, you should either increase 
query_mem_limit
or lower the 
pipeline_dop
. References • sql-reference/System_variable.md • administration/management/resource_management/Memory_management.md • faq/resource_isolation_faq.md • administration/management/resource_management/Query_management.md • faq/Sql_faq.md • administration/management/resource_management/query_queues.md
r
@Rocky On a 100-CN cluster with pipeline_dop=4, does a single large query automatically spread its fragment instances across multiple CNs so the effective parallelism is 4 × 100 = 400 drivers, or is there a per-query cap (e.g., fragment_instance_num_per_query)?
r
On a 100-CN cluster, the StarRocks scheduler will naturally attempt to parallelize the query across as many nodes as necessary to process the data, but the "effective parallelism" is governed by how data is distributed and specific "cap" variables. 1. Does it spread across all 100 CNs? Yes, for the Scan Fragment. If your data (table tablets/partitions) is distributed across all 100 CNs, the query's first fragment (the Scan Fragment) will automatically spawn instances on every node that holds a piece of that data. * Default Behavior: If the table is large and spread across the whole cluster, the scan fragment will run on all 100 CNs. * Parallelism Formula: On each CN, the number of instances is set by 
parallel_fragment_exec_instance_num
(default is 1). Each of those instances then runs 
pipeline_dop
drivers. * Result: $100 \text{ (Nodes)} \times 1 \text{ (Instance/Node)} \times 4 \text{ (DOP)} = \mathbf{400}$ concurrent drivers across the cluster for that specific fragment. 2. Is there a Per-Query Cap? Yes, there are two primary caps that prevent a single query from taking over a massive cluster or being limited to a tiny subset: * 
max_parallel_scan_instance_num
(The "Total Instances" Cap): Introduced in v3.2, this session variable caps the total number of scan instances the planner will produce cluster-wide for a single query. * Default (-1): No hard cap; it uses as many nodes as have data. * If set to 50: Even if your data is on 100 nodes, the scheduler will only use 50 nodes (or 50 instances) to perform the scan, forcing nodes to process more data per instance to stay under the cap. * 
parallel_exchange_instance_num
(The "Intermediate" Cap): This controls the parallelism of "Exchange" fragments (shuffles, joins, aggregations). * Default (-1): It matches the parallelism of the lower-level scan fragment. * Manual Control: You can set this to a lower value (e.g., 20) to ensure that while the scan happens on 100 nodes, the complex aggregation "merging" only happens on 20 nodes to reduce network shuffle overhead. 3. Effective Parallelism Summary For a single query on a 100-CN cluster with `pipeline_dop=4`: Scenario | Total Cluster-Wide Drivers | Why? Small Table (Data on 10 nodes) | $10 \times 1 \times 4 = \mathbf{40}$ | Limited by data distribution. Large Table (Default settings) | $100 \times 1 \times 4 = \mathbf{400}$ | Spreads to all nodes holding data. With 
max_parallel_scan_instance_num=50
 | $50 \times 1 \times 4 = \mathbf{200}$ | Capped by the planner's instance limit. Summary There is no automatic "10-node limit" for queries; StarRocks is designed to scale linearly. However, if you want to prevent a single query from "exploding" into thousands of drivers on a 100-node cluster, you should use 
max_parallel_scan_instance_num
to restrict the total number of fragment instances the FE will schedule. Rocky's Tip: You can verify how many nodes and instances were used for any query by checking the 
Query Profile
. Look for the "InstanceNum" and "BackendNum" metrics in the Fragment section of the profile. References • sql-reference/System_variable.md • best_practices/query_tuning/query_planning.md • administration/management/resource_management/Query_management.md • faq/Sql_faq.md • best_practices/query_tuning/query_profile_operator_metrics.md References • sql-reference/System_variable.md • faq/resource_isolation_faq.md
r
@Rocky Query Queue (v3.1.4+) calculates slots based on pipeline_dop. If a session manually overrides SET pipeline_dop=16 on a CN that defaults to auto(4), does Query Queue recalculate slot usage live, or does it keep the original slot estimate and cause overcommit?
r
In StarRocks (v3.1.4+), the Query Queue system and the "Dynamic DOP Adjustment" mechanism work together to prevent the exact overcommit scenario you are describing. Here is how the accounting and enforcement model works when you manually override `SET pipeline_dop=16`: 1. Does it recalculate "live"? Yes, but specifically at query submission time. The "slots" or "drivers" are not estimated once and forgotten; the FE (specifically the Leader FE) performs a fresh resource check for every incoming query. * Step 1 (Manual Setting): When you 
SET pipeline_dop=16
, the FE's planner initially intends to use 16 drivers per fragment instance. * Step 2 (Query Queue Check): Before execution, the FE calculates the query's resource footprint: 
num_drivers = num_fragments × 16
. * Step 3 (The Safety Check): The FE compares this against the cluster-wide 
query_queue_driver_high_water
(the "High Water Mark"). * Step 4 (Adjustment): If running with 
pipeline_dop=16
would push the BE's total driver count over the high water mark, the FE automatically scales down your manual override for that specific query execution (e.g., it might force it to 8 or 4) to stay within the cluster's healthy limits. 2. Does it cause overcommit? No. The Query Queue does not "blindly" trust the default setting. It uses the Effective DOP for its accounting. * If the FE allows the 16 DOP: It will subtract "16 drivers per fragment" from the available capacity in the Query Queue. Other incoming queries will then see less available "room" and will be queued. * If the FE forces it down: It will only subtract the amount it actually allows (e.g., 4 drivers) and execute the query with that lower parallelism. In either case, the system remains "resource-aware." There is no scenario where the system thinks you are using 4 drivers (the default) while you are actually using 16. 3. Slot Accounting (Query Queue V2 / v3.3.4+) In newer versions, StarRocks uses a "Slot" model (
enable_query_queue_v2 = true
). * Total Slots: The cluster's total capacity is calculated as 
Total Slots = query_queue_v2_concurrency_level × Number of BEs × Cores per BE
. * Query Slot Usage: If you have 
query_queue_slots_estimator_strategy = PBE
(Parallelism-Based Estimation), the query will consume slots proportional to its DOP. * The Result: A query with 
pipeline_dop=16
will simply consume 4x more slots than a query with 
pipeline_dop=4
. If there aren't enough slots available to satisfy the "16-DOP" request, the query will sit in the queue until enough slots are freed by finishing queries. Summary * Manual Overrides: Are respected by the planner but vetted by the Query Queue. * Recalculation: Happens for every query based on the session's current 
pipeline_dop
. * Overcommit: Is prevented because the FE is the gatekeeper—it either accounts for the high DOP by taking more "slots" or it forces the DOP down to protect the BEs. Rocky's Advice: While you can manually override DOP, it is usually safer to let StarRocks handle it by keeping 
pipeline_dop=0
(Auto). If you must override it for a specific heavy report, the Query Queue acts as your "insurance policy" to ensure that one session's 
DOP=16
doesn't crash the BEs for everyone else. References • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md • administration/management/resource_management/Query_management.md References
r
@Rocky When enable_pipeline_engine=true (default), is parallel_fragment_exec_instance_num completely ignored, or does it still affect certain fragment types (e.g., exchange, hash-join build side, broadcast exchange)? Any case where both variables still interact?
r
When 
enable_pipeline_engine=true
(which has been the default since StarRocks 2.3), the relationship between these two variables is as follows: 1. Is 
parallel_fragment_exec_instance_num
ignored? Yes, for almost all standard query compute fragments. In the modern Pipeline engine, StarRocks has transitioned from "Instance-level parallelism" to "Driver-level parallelism." * The Old Way: Parallelism was achieved by spawning multiple Fragment Instances on a single BE. If you set 
parallel_fragment_exec_instance_num=8
, the BE would run 8 separate instances of the same fragment. * The Pipeline Way: StarRocks creates one Fragment Instance per BE for each fragment and achieves parallelism via Pipeline Drivers (controlled by 
pipeline_dop
). To prevent a "parallelism explosion" (where you might accidentally get 8 instances × 8 drivers = 64 threads for a single query), the FE Planner automatically forces 
parallel_fragment_exec_instance_num
to 1 when the pipeline engine is active. 2. Are there exceptions? While ignored for compute (Joins, Aggregates, Scans in SELECTs), there are specific "edge cases" or legacy interactions: * Data Sinks (INSERT / Loading): For 
INSERT INTO ... SELECT
or Broker Loads, StarRocks may still use multiple fragment instances to improve sink throughput. In these cases, the FE might use 
load_parallel_instance_num
or honor 
parallel_fragment_exec_instance_num
to spawn multiple "Sink" instances on a BE so they can write to different tablets in parallel. * External Table Scans: In some earlier 2.x/3.x versions, certain external table connectors (like HDFS/S3 scans) occasionally used multiple instances to manage I/O concurrency before the "Scan Operator" became fully adaptive, though this has largely been consolidated into 
pipeline_dop
and 
connector_scan_node_number
in recent versions. * Broadcast & Exchange: These do not use 
parallel_fragment_exec_instance_num
. The number of Exchange receivers is instead controlled by 
parallel_exchange_instance_num
(default -1, which matches the number of instances—usually 1 per BE). 3. Do the variables still interact? They do not multiply. If you manually set 
SET parallel_fragment_exec_instance_num = 16
and 
SET pipeline_dop = 8
, the system will ignore the 16 and run the query with 1 instance and 8 drivers per BE. Summary Comparison Variable | Pipeline Engine = TRUE | Pipeline Engine = FALSE Primary Parallelism | 
pipeline_dop
| 
parallel_fragment_exec_instance_num
Unit of Execution | Pipeline Driver (Threads) | Fragment Instance (Processes/Contexts) Instance Count | Forced to 1 (usually) | User-defined Benefit | Lower overhead, better scheduling | Simple but higher memory/context overhead Rocky's Advice: If you are on StarRocks 3.x+, you should stop using 
parallel_fragment_exec_instance_num
entirely. If you need more speed, increase 
pipeline_dop
or set it to 
0
(Auto). The only time you should look at "Instance" variables is if you are debugging a very specific 
INSERT INTO
performance issue where the sink is the bottleneck. References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • faq/Sql_faq.md • best_practices/query_tuning/query_planning.md References • knowledge/usage/best-practice.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md
r
@Rocky What is the "morsel" size in StarRocks Morsel-Driven execution? Is it configurable (chunk_size, vectorized_batch_size)? How does morsel size interact with pipeline_dop — does a larger morsel reduce effective DOP?
r
In the StarRocks Pipeline Engine, Morsel-Driven Parallelism is the mechanism used to achieve dynamic load balancing by breaking down large scan tasks into smaller, schedulable units called Morsels. 1. What is the "Morsel" size? In StarRocks, a "morsel" is the basic unit of a scan task. Unlike a "Tablet" (which is a physical data distribution unit), a morsel is a runtime execution unit. * Default Definition: By default, StarRocks tries to split scan data into morsels of roughly 64,000 to 128,000 rows (internally often 64k rows). * Context: While a "Tablet" might contain 1 GB or 10 million rows, the scheduler breaks that tablet into many "Morsels" so that multiple CPU cores (drivers) can work on the same tablet simultaneously. 2. Is it configurable? Morsel size is generally an internal heuristic, but it is influenced by several related parameters: * 
vector_chunk_size
(Default: 4096): This is the size of the Chunk (vectorized batch) that flows between operators (e.g., from Scan to Filter). It is the unit of vectorized processing, whereas a morsel is the unit of scheduling. One morsel typically contains multiple chunks. * 
scanner_row_num
(BE Config, Default: 16384): This controls the maximum number of rows a single scan thread fetches from the storage engine at once. * 
enable_tablet_internal_parallel
(Default: true): This is the "on/off" switch for morsel-splitting. If set to 
false
, 1 Tablet = 1 Morsel. If 
true
, a tablet is split into many small morsels. * 
query_queue_v2_num_rows_per_slot
(Default: 4096): Used in Query Queue v2 to estimate how many "slots" (parallel resources) a query needs based on its scanned row count. 3. Interaction with 
pipeline_dop
 The morsel size and 
pipeline_dop
have a direct relationship regarding Effective Parallelism: * The Parallelism Cap: The effective degree of parallelism for a scan is limited by the number of available morsels. $$\text{Effective DOP} = \min(\text{Number of Morsels}, \text{pipeline\_dop})$$ * Does a larger morsel reduce effective DOP? Yes. If your morsel size is too large (or if your data is very small), you may end up with fewer morsels than you have available CPU threads (
pipeline_dop
). _ _Example:* You have 1 million rows and 
pipeline_dop=64
. If the morsel size is 1 million rows (no splitting), only 1 thread will work while 63 sit idle. _ _StarRocks Solution:* By splitting that 1 million rows into 100 morsels (each ~10k rows), StarRocks ensures that all 64 drivers have work to pull from the queue, achieving the full 
pipeline_dop
. Summary Concept | Size / Unit | Purpose Morsel | ~64k - 128k Rows | Unit of Scheduling (Work task for a driver). Chunk | 4096 Rows | Unit of Processing (SIMD/Vectorized batch). Tablet | ~1 GB / Millions of Rows | Unit of Storage (Physical distribution). Rocky's Tip: You can see how many morsels were generated for your query by checking the 
MorselsCount
metric in the 
OLAP_SCAN_OPERATOR
section of the Query Profile. If 
MorselsCount
is significantly lower than 
pipeline_dop
, you are likely bottlenecked by data volume or tablet count, and your effective DOP will be lower than the setting. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md
r
@Rocky Is SIMD vectorization in StarRocks operators orthogonal to pipeline_dop? That is, with pipeline_dop=1 on a single thread, is per-batch SIMD still active, or does Pipeline Engine parallelism also gate SIMD code paths?
r
Yes, SIMD (Single Instruction, Multiple Data) vectorization in StarRocks is completely orthogonal to 
pipeline_dop
. If you set 
pipeline_dop=1
, the Pipeline Engine parallelism is gated to a single thread, but the per-batch SIMD instructions remain fully active. Here is the technical breakdown of why these two systems do not interfere: 1. Layers of Parallelism StarRocks achieves performance through two distinct layers: * Vectorized Execution (SIMD): This is Data-Level Parallelism. It happens inside a single operator (e.g., a Filter or an Aggregate). When an operator processes a "Chunk" (usually 4,096 rows), it uses SIMD instructions (like AVX2 or AVX-512) to process multiple rows in that chunk simultaneously within a single CPU core. * Pipeline Engine (
pipeline_dop
): This is Task-Level Parallelism. It handles how many CPU cores are assigned to run those operators. 2. The Single-Threaded Scenario (
pipeline_dop=1
) When you set `pipeline_dop=1`: 1. The FE scheduler assigns exactly one Pipeline Driver (one execution thread) to the query fragment. 2. That thread pulls a Chunk of data from the source (e.g., an OLAP scan). 3. Even though there is only one thread, that thread calls the Vectorized Operators. 4. Inside those operators, the code is compiled to use SIMD. For example, if it's adding two columns, it will still use a SIMD instruction to add 8 or 16 integers at once. SIMD is not "gated" or disabled by low DOP. The only way to disable SIMD would be to use a CPU that doesn't support the instruction set or to use a non-vectorized execution engine (which StarRocks phased out years ago). 3. Key Variables for Each Layer Feature | Controlling Variable | Unit of Work Vectorization / SIMD | 
vector_chunk_size
(Default 4096) | A Chunk (Batch of rows) Multi-core Parallelism | 
pipeline_dop
| A Pipeline Driver (Thread) Summary * Is it orthogonal? Yes. * Does 
pipeline_dop=1
still use SIMD? Yes. The single thread will still process data in 4,096-row batches using SIMD instructions. * Does Pipeline Engine gate SIMD? No. The Pipeline Engine only manages how many threads are running; it does not change the internal logic of the operators those threads are executing. Rocky's Tip: If you are trying to debug performance, setting 
pipeline_dop=1
is a great way to isolate whether a bottleneck is due to computational complexity (SIMD/Vectorization efficiency) versus scheduling overhead/contention (Pipeline efficiency). References • administration/management/BE_configuration.md • faq/resource_isolation_faq.md • faq/Sql_faq.md • best_practices/query_tuning/query_planning.md • knowledge/usage/best-practice.md • sql-reference/System_variable.md
r
@Rocky In EXPLAIN VERBOSE output, which field shows the planned driver count per fragment so we can predict DOP before execution? And in ANALYZE PROFILE / pipeline_profile_level=1, which field reports the actual driver count executed?
r
To predict and verify the degree of parallelism (DOP) in StarRocks, you should look for the following fields in your query plans and profiles. 1. In 
EXPLAIN VERBOSE
(Planned DOP) In the output of 
EXPLAIN VERBOSE
, StarRocks displays the planned parallelism for each Plan Fragment. You need to look at the header of each fragment (e.g., 
PLAN FRAGMENT 0
). * Field: 
Pipeline Dop
 (or sometimes listed as 
pipeline_dop
depending on the version/UI). * Location: It is usually found right at the top of each fragment definition, near the 
Partition
and 
Instance Num
fields. * What it represents: This is the number of Pipeline Drivers the FE (Frontend) planner has decided to spawn for this specific fragment on each Backend (BE). If 
pipeline_dop
is set to 
0
(the default), you will see the adaptively calculated value here (e.g., 
Pipeline Dop: 16
). ────────── 2. In 
ANALYZE PROFILE
/ 
pipeline_profile_level=1
(Actual DOP) When you analyze a profile (using 
ANALYZE PROFILE FROM '<query_id>'
or by examining a text-based profile), you are looking for the actual execution count of drivers. * Field: 
DegreeOfParallelism
 * Location: In the Pipeline level of the profile tree. * If you use 
ANALYZE PROFILE
, look at the nodes labeled 
Pipeline
. * In a raw text profile, it is listed under the 
Pipeline (id=X)
section. * What it represents: This is the number of Pipeline Drivers that actually ran for that pipeline on a single BE node. * Related Field: 
TotalDegreeOfParallelism
: This is the aggregate sum of all drivers across all participating BE nodes for that specific pipeline. (e.g., if you have 3 BEs and 
pipeline_dop=8
, 
DegreeOfParallelism
will be 
8
and 
TotalDegreeOfParallelism
will be 
24
). Summary Table Goal | Tool / Command | Target Field | Level Predict DOP | 
EXPLAIN VERBOSE <sql>
| 
Pipeline Dop
 | Fragment Header Verify DOP | 
ANALYZE PROFILE FROM '...'
| 
DegreeOfParallelism
 | Pipeline Node Cluster-wide DOP | 
ANALYZE PROFILE FROM '...'
| 
TotalDegreeOfParallelism
 | Pipeline Node Rocky's Tip: If you notice that 
DegreeOfParallelism
in the profile is lower than the 
Pipeline Dop
in the 
EXPLAIN
plan, it usually indicates that StarRocks' adaptive scheduler reduced parallelism due to low data volume (e.g., the number of morsels/tablets was smaller than the requested DOP) or resource group constraints. References • best_practices/query_tuning/query_profile_operator_metrics.md • sql-reference/System_variable.md • knowledge/usage/best-practice.md
r
@Rocky Does StarRocks have any published benchmark or internal guidance comparing pipeline_dop=0/2/4/8/12 on the same query with fixed 8-core CN? We want to understand the diminishing return curve beyond pipeline_dop=cores/2.
r
While StarRocks does not provide a single "official" graph for every hardware permutation, the behavior of the Pipeline Engine across different 
pipeline_dop
settings on a fixed 8-core node follows a well-understood performance curve based on scheduling overhead and resource contention. 1. The Scaling Curve (8-Core CN) Based on internal guidance and common benchmarking patterns (like TPC-H/TPC-DS), here is how performance typically scales on an 8-core machine: Setting | Strategy | Performance Trend 
pipeline_dop=1
| High Concurrency | Best for high QPS environments. Minimizes scheduling overhead and maximizes the number of simultaneous queries the cluster can handle. 
pipeline_dop=4
| Standard (0/Default) | Sweet Spot. StarRocks defaults to 
cores/2
. This provides a significant speedup for single queries while leaving "headroom" for the OS, I/O threads, and background tasks (compaction). 
pipeline_dop=8
| Peak Throughput | Near-maximum speed for a single large query. You will see diminishing returns here compared to the jump from 2 to 4 because the CPU becomes fully saturated, leaving no room for other system tasks. 
pipeline_dop=12
| Over-subscription | Negative Returns. Since you only have 8 physical cores, 12 drivers will force the OS to perform frequent context switching. You will likely see higher latency and increased CPU "System" time compared to 
pipeline_dop=8
. 2. Why the "Diminishing Return" occurs The diminishing return beyond 
cores/2
(DOP=4) and the regression beyond 
cores
(DOP=8) are driven by three main factors: * Morsel Scheduling Overhead: The Pipeline Engine is a pull-based scheduler. As you increase DOP, more threads are competing for the "Global Morsel Queue." If the morsel size is small, the overhead of threads constantly grabbing new work can outweigh the parallel processing gains. * Cache Locality: StarRocks is highly optimized for CPU cache. When 
pipeline_dop
exceeds the physical core count, multiple threads are swapped in and out of the same core, causing L1/L2 cache misses that degrade the performance of vectorized operators. * Resource Contention: Beyond the execution threads, StarRocks has background threads for: * I/O Scanners: Fetching data from disk/S3. * Network: Sending/receiving shuffle data. * Metadata: FE-BE heartbeats and reporting. Setting 
pipeline_dop=cores
effectively "starves" these helper threads, which can lead to I/O bottlenecks. 3. Internal Logic for 
pipeline_dop=0
 In recent versions (v3.1+), the "Adaptive" setting (
0
) is more sophisticated than just a fixed division: 1. Lower Bound: It calculates a base DOP (usually 
physical_cores / 2
). 2. Adaptive Cap: It considers the Resource Group limits. If a user is restricted to 25% CPU, 
pipeline_dop=0
will cap the driver count to stay within that quota. 3. Fragment Complexity: For very simple queries (e.g., 
SELECT * FROM table LIMIT 10
), it may automatically reduce the effective DOP to 1 to save the cost of spawning and destroying drivers. Recommendation for 8-Core CN * For General Workloads: Keep 
pipeline_dop=0
. It will likely settle at 4, providing the best balance between single-query speed and system stability. * For Latency-Critical Batch: If you are running a single massive ETL job at night, try 
pipeline_dop=8
. * Avoid `pipeline_dop > 8`: There is almost no scenario on an 8-core machine where DOP 12 will outperform DOP 8; it will only increase CPU jitter and memory pressure. Rocky's Tip: Use 
ANALYZE PROFILE
to check the 
CPUTime
vs. 
WallTime
ratio. If 
WallTime
is significantly higher than 
CPUTime / DOP
, it means your drivers are spending too much time waiting for resources (morsels or I/O), indicating your DOP is set too high. References • knowledge/usage/best-practice.md
r
@Rocky Is there any interaction between pipeline_dop and Data Cache access? For example, does higher DOP create contention on the same cached data files (sstable-level lock), or are Data Cache reads fully concurrent?
r
The interaction between 
pipeline_dop
and StarRocks' Data Cache (also known as Block Cache or StarCache) is designed to be highly concurrent, but there are specific architectural nuances regarding how threads interact with the cache layers. 1. Are Data Cache Reads Fully Concurrent? Yes, for the most part. The Data Cache is a block-level cache that splits data into fixed-size chunks (typically 1MB). It does not use a "coarse-grained" or "sstable-level" lock that would serialize reads from the same file. * Block-Level Granularity: Since the cache key is a combination of 
hash(file_path) + block_id
, different pipeline drivers (threads) reading different parts of the same file are accessing entirely different entries in the cache's metadata and storage. * Sharded Metadata Locks: To handle high concurrency (high 
pipeline_dop
), StarRocks shards its cache metadata (the hash table that tracks which blocks are in memory/disk). Instead of one global lock, it uses thousands of "lock shards." This ensures that multiple drivers can query the cache index simultaneously with negligible contention. * Lock-Free Read Path: Once a driver identifies that a block is present (a cache hit), the actual data read from the local medium (NVMe/SSD or RAM) is performed without holding an execution-blocking lock on the file metadata. 2. Does High DOP Create Contention? While there is no "sstable-level lock," increasing 
pipeline_dop
to extreme levels (e.g., beyond the number of physical cores) can introduce three types of "soft" contention: * SSD/NVMe I/O Queue Contention: If 
pipeline_dop
is very high, many drivers may issue 
pread
calls to the same physical disk simultaneously. While NVMe handles high parallelism well, you can eventually hit the disk's maximum IOPS or queue depth. This is visible in the query profile as an increase in 
DataCacheReadTimer
. * Morsel-to-Cache Mapping: Each pipeline driver processes a "morsel" of data. If your DOP is higher than the number of available morsels, some drivers will sit idle. If many drivers try to "populate" the same missing block simultaneously (a cache miss), StarRocks uses Write-Coalescing. One thread will perform the remote fetch while the others wait/subscribe to the result, preventing redundant remote I/O. * Page Cache Contention: If the data is being served from the Operating System Page Cache (the "Memory" part of the Data Cache), high DOP can lead to CPU cache line bouncing in the kernel's memory management code, though this is rarely a bottleneck compared to disk I/O. 3. Sstable-Level Logic vs. Data Cache The term "sstable-level lock" usually refers to LSM-tree compaction or row-set merging. In StarRocks: * For Shared-Data (Cloud-Native): The Data Cache sits below the storage engine logic. The 
OlapScanNode
requests specific ranges of a segment file; the Data Cache simply provides those bytes. It is blissfully unaware of "row-sets" or "versions" at this layer, which is why it's so fast. * For Local Storage (Shared-Nothing): StarRocks uses its own columnar segment format. Concurrent reads to the same segment file are handled via 
pread
(positional read), which is thread-safe and lock-free at the OS level. Summary Level | Concurrency Mechanism | Contention Risk File Level | No locks; uses positional offsets (
pread
). | None (at the engine level). Cache Metadata | Sharded Hash Table Locks. | Extremely low (unless DOP > 128). Cache Data | Parallel I/O (NVMe Queue). | Medium (Hardware-limited). Cache Miss | Write-Coalescing (One thread fetches, others wait). | Low (Prevents redundant S3/HDFS calls).
Rocky’s Recommendation: If you see 
DataCacheReadTimer
spiking when you increase 
pipeline_dop
, it’s likely not a software lock but physical I/O saturation on your local SSDs. In such cases, adding more NVMe drives or increasing the BE memory (to leverage the OS Page Cache) is more effective than tuning DOP tuning. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md • deployment/shared_data/feature-support-shared-data.md • sql-reference/System_variable.md • using_starrocks/caching/query_cache.md References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • using_starrocks/caching/query_cache.md • data_source/data_cache_troubleshooting.md • data_source/data_cache_observe.md
r
@Rocky connector_scan_node_number as a modern replacement for per-instance controls on external table scans. On a 100-CN cluster with Iceberg external catalog and plan_mode=distributed, what is the default value of connector_scan_node_number, and how does it interact with pipeline_dop — is it a hard cap on the number of CNs used for scan, or a per-CN instance multiplier?
r
In StarRocks (v3.3+), 
connector_scan_node_number
is indeed the modern session variable designed to control the horizontal scaling of external table scans (Iceberg, Hive, Hudi, etc.), complementing the vertical scaling provided by 
pipeline_dop
. 1. Default Value * Default: 
0
* Meaning: When set to 
0
, StarRocks uses all available healthy Compute Nodes (CNs) or Backends (BEs) in the cluster that are eligible for the scan. * On a 100-CN cluster: By default, a large Iceberg query will attempt to use all 100 nodes to maximize throughput, provided the number of data files/splits is sufficient to justify the distribution. 2. Interaction with 
pipeline_dop
 These two variables control different dimensions of parallelism: * 
connector_scan_node_number
(Horizontal): This is a hard cap on the number of nodes participating in the scan. If you have 100 CNs but set 
connector_scan_node_number = 20
, StarRocks will only schedule the scan fragment on 20 nodes. * 
pipeline_dop
(Vertical): This defines the number of Pipeline Drivers (parallel threads) executed per node. The Formula: $$\text{Total Scan Drivers} = \min(\text{Available Nodes}, \text{connector\_scan\_node\_number}) \times \text{pipeline\_dop}$$ 3. Interaction with 
plan_mode=distributed
 When querying Iceberg with 
plan_mode=distributed
, StarRocks offloads the manifest file parsing and metadata retrieval from the FE to the CNs/BEs. * Metadata Phase: 
connector_scan_node_number
also acts as a limit for the number of nodes used to perform the distributed metadata planning. * Data Phase: Once planning is complete, the same limit applies to the actual data file scanning. * Efficiency: For a 100-CN cluster, if you are running many concurrent small-to-medium queries, setting 
connector_scan_node_number
to a value like 
16
or 
32
is highly recommended. This prevents the "scatter-gather" overhead where 100 nodes each process only a handful of rows, which often leads to higher latency due to network shuffle and scheduling tail-latency. Summary Table for 100-CN Cluster Variable | Recommended for Large Scans | Recommended for High Concurrency | Role 
connector_scan_node_number
 | 
0
(All 100 nodes) | 
8
or 
16
| Hard Cap on total participating nodes. 
pipeline_dop
 | 
0
(Adaptive, usually 1/2 cores) | 
2
or 
4
| Parallelism within each single node. Rocky's Tip: If you see high 
QueryAllocatedTimer
or 
NetworkEthernetTimer
in your profiles on a 100-node cluster, it's often a sign of over-distribution. Try capping 
connector_scan_node_number
to a smaller subset of the cluster to improve cache hit rates (Data Cache) and reduce coordination overhead. References • sql-reference/System_variable.md • administration/management/BE_configuration.md • using_starrocks/Cost_based_optimizer.md
r
@Rocky enable_tablet_internal_parallel=true splits a tablet into multiple morsels. On a shared_data (Iceberg/S3) setup where there is no native "tablet" concept, what is the equivalent unit that morsel splitting operates on (file-split? row-group?), and does enable_tablet_internal_parallel have any effect, or is there a separate variable like enable_connector_split_parallel?
r
In StarRocks, the behavior of internal parallelism differs significantly between internal tables (OLAP/Lake) and external tables (Iceberg/S3/Hive). 1. Does 
enable_tablet_internal_parallel
affect Iceberg? No. The variable 
enable_tablet_internal_parallel
(and its cloud-native sibling 
enable_lake_tablet_internal_parallel
) is explicitly designed for StarRocks' own storage format. It allows a single tablet (which consists of multiple segments) to be split into multiple morsels so that multiple threads can scan different segments of that tablet simultaneously. Since Iceberg and other external tables do not use the "tablet" concept, these variables have no effect on them. 2. The Equivalent: File Splits For external catalogs (Iceberg, Hive, Hudi), the equivalent unit that "morsel splitting" operates on is the Split. * The Unit: For Parquet/ORC files in Iceberg, a "split" is typically a File Range or a Row Group. StarRocks splits large files into multiple ranges (usually 128MB by default, or aligned with Parquet Row Groups) during the planning phase. * The Morsel: In the Pipeline Engine, each one of these Splits is treated as a single "morsel." * Parallelism Mechanism: If you have an Iceberg table with 1 large Parquet file that has 8 Row Groups, StarRocks will generate 8 Splits. If your 
pipeline_dop
is 8, all 8 Row Groups can be processed in parallel by 8 different Pipeline Drivers on that node. 3. Relevant Variables for External Scan Parallelism Instead of 
enable_tablet_internal_parallel
, external scans are tuned using these controls: Variable | Role for External Tables 
pipeline_dop
 | Determines how many Pipeline Drivers (threads) can pull and process splits (morsels) simultaneously on a single CN. 
connector_scan_node_number
 | (As discussed earlier) Caps the number of CNs that can participate in the scan. 
enable_connector_adaptive_io_tasks
 | Enabled by default (
true
). It allows the scan operator to dynamically manage I/O concurrency based on system load. 
connector_io_tasks_per_scan_operator
 | The maximum number of concurrent I/O requests (e.g., S3 GET requests) a single scan operator can have in flight. The default is 16. 4. Why there is no "Internal Parallel" variable for Connectors In internal OLAP tables, "tablets" are a hard storage boundary. If a user only created 1 bucket (1 tablet), without 
enable_tablet_internal_parallel
, they could only use 1 thread to scan that data regardless of how many cores they had. In contrast, External Connectors are "Split-Native." The StarRocks FE (or CN in 
distributed
plan mode) automatically performs split-calculating logic to ensure there are enough splits to saturate the available 
pipeline_dop
. If a file is large enough, it is already broken down into multiple morsels (splits) before it ever reaches the execution engine. Summary * Internal Tables: Use 
enable_tablet_internal_parallel
to break 1 tablet into $N$ morsels. * Iceberg/External: Use the Split mechanism (automatic) to break 1 file into $N$ morsels. * Recommendation: If you feel an Iceberg query is under-utilizing your 8-core CN, check the 
ScanRangesNum
in the query profile. If it is high, increase 
pipeline_dop
. If it is low (e.g., only 1 split), the bottleneck is the number of files or the row-group size in your source Iceberg data. Rocky's Tip: For Iceberg, ensure you are using 
plan_mode=distributed
. This allows the CNs to help split the manifest files into data splits in parallel, which is much faster than the FE doing it alone for large datasets. References • sql-reference/System_variable.md • data_source/catalog/iceberg/iceberg_catalog.md • administration/management/BE_configuration.md
Open in Slack
PreviousNext
Powered by