@Rocky After deploying a 4.0 cluster, im having performance differences with a 3.5 cluster as concurrency increases. What configs might be causing performance degradation after an upgrade

In StarRocks 4.0, several architectural shifts and default configuration changes may affect performance under high concurrency compared to 3.5. Here are the primary areas to investigate for performance differences after an upgrade: 1. Execution Engine & Parallelism StarRocks 4.0 introduced a redesigned execution engine with improved parallelization. * Pipeline DOP (

pipeline_dop

): By default, this is

0

(adaptive). In high-concurrency scenarios, the system might over-allocate CPU resources to single queries, starving others. * Action: Try setting

SET GLOBAL pipeline_dop = 1;

or a low fixed value to see if it stabilizes performance by reducing resource contention per query. * Shared Scan: In some 4.0 patches (notably v4.0.1),

shared_scan

was temporarily disabled due to scheduling changes. If your 3.5 workload relied on scanning the same tables/partitions across many concurrent queries, this loss of optimization could significantly increase I/O. 2. Cache Configuration Changes StarRocks 4.0 simplified Data Cache and Page Cache management, deprecating several old parameters. * Deprecated Parameters:

storage_page_cache_limit

,

block_cache_mem_size

, and

block_cache_disk_size

are no longer the primary configs. * New Parameters: Ensure you are using

datacache_mem_size

and

datacache_disk_size

. If you haven't migrated your 3.5 tuning to these new parameters, the cluster may be running with default (and potentially smaller) cache sizes. * Metadata Cache Limits: For external catalogs (Iceberg/Hive), 4.0 introduced

remote_file_cache_memory_ratio

and

iceberg_data_file_cache_memory_usage_ratio

, defaulting to 10%. If your metadata is large, this might cause more frequent remote fetches than in 3.5. 3. Automatic Bucketing The default bucket size for automatic bucket tables changed from 4GB to 1GB in v4.0. * Impact: While 1GB buckets generally improve parallelization, they increase the total number of tablets and metadata overhead. If your cluster has a very high number of tablets, the FE and BE might experience higher overhead under heavy concurrency. 4. Query Queue & Resource Management StarRocks 4.0 relies more heavily on Query Queue v2, which uses logical "slots" based on estimated resources. * Queuing Delay: Check

SHOW PROCESSLIST

to see if queries are marked as

IsPending

. If the slot estimation is too conservative, queries may be queuing even when the system seems to have idle resources. * Concurrency Limits: Verify that

query_queue_concurrency_limit

and your Resource Group

concurrency_limit

are not set too low. 5. Data Lake & Connector Tuning If you are querying external tables (Data Lakehouse): * Adaptive I/O:

enable_connector_adaptive_io_tasks

is

true

by default. Under high concurrency, it can sometimes spawn too many I/O tasks. * Action: Try setting

set global enable_connector_adaptive_io_tasks = false;

and manually limit

connector_io_tasks_per_scan_operator

(e.g., to 4 or 8). 6. Network & RPC For high concurrency, StarRocks often recommends tuning the communication layer: * BRPC Connection: Set the BE configuration

brpc_connection_type=pooled

to reduce RPC latency and connection overhead under high QPS. Recommended Troubleshooting Steps 1. Compare Profiles: Use

SET enable_profile = true;

on both clusters and compare the

QueryProfile

. Look for operators with high

WaitTime

or significantly higher CPU usage in 4.0. 2. Check Audit Logs: Monitor the

QueryTime

vs.

WaitTime

in the FE audit logs to determine if the delay is in execution or queuing. 3. System Configs: Ensure your OS-level

max_map_count

is set to at least

262144

, as 4.0's new engine may create more memory map areas. References • deployment/shared_data/feature-support-shared-data.md

@Sebastian Garcia check query queue v2

Looks to be enabled on both versions, but ill review related settings. Thanks !

false

(v3.3 to v4.0)

true

(from v4.1 onwards) since you are on 4.0 i think my comment is noise, sorry

@Rocky what are the implications for resources and performance when setting

pipeline_dop = 1;

Setting

pipeline_dop = 1

forces the execution engine to use exactly one thread (pipeline instance) per query fragment on each BE node. In StarRocks 4.0, the default is

0

(adaptive), which usually targets half of the available CPU cores. Transitioning to

1

has several specific implications: 1. Performance Implications * Reduced Single-Query Latency: For a single, large OLAP query, performance will significantly decrease. Since the query cannot parallelize across cores on a single node, operations like scans, joins, and aggregations will take longer. * Increased System Throughput (Concurrency): This is the primary reason to use

pipeline_dop = 1

. By preventing any single query from "hogging" all CPU cores, the system can handle a much higher volume of concurrent queries (QPS). It reduces thread context-switching and resource contention at the OS level. * Predictability: In high-concurrency environments (e.g., thousands of point lookups or simple ODS queries), setting this to 1 makes query performance more stable and prevents "noisy neighbor" scenarios where one complex query spikes CPU and slows down all others. 2. Resource Implications * CPU Utilization: * Per-Query: Much lower. A single query can at most saturate one core per fragment. * Global: Often more efficient for high-concurrency workloads. Adaptive parallelism (

dop=0

) can sometimes over-parallelize simple queries, leading to diminishing returns where more CPU time is spent on scheduling overhead than on actual data processing. * Memory Usage: * Setting

pipeline_dop = 1

typically lowers memory consumption per query. Many operators (like Hash Joins or Aggregations) create local buffers or hash tables for every parallel instance. Reducing parallelism from, say, 16 to 1 drastically reduces the number of these data structures residing in memory at once. * I/O Pattern: * It results in more sequential-like I/O per query. While this might be slower for a single query on NVMe drives, it can prevent I/O thrashing in environments where many concurrent queries are already saturating the disk bandwidth. Comparison Summary Metric |

pipeline_dop = 1

|

pipeline_dop = 0

(Adaptive/High) Best For | High QPS, simple queries, point lookups | Complex OLAP, ETL, large-scale aggregation Single-Query Speed | Slow | Fast Max Concurrency | Very High | Lower (Resource limited) Memory Overhead | Minimal | High (scales with DOP) CPU Efficiency | High (low overhead) | Varies (scheduling overhead at high DOP) Recommendation for 4.0 If you are seeing degradation as concurrency increases, try setting

SET GLOBAL pipeline_dop = 1;

. This is a common "best practice" for high-concurrency serving scenarios to ensure the 4.0 scheduler doesn't over-allocate cores to individual requests at the expense of overall throughput. References • knowledge/usage/best-practice.md • sql-reference/System_variable.md • faq/Sql_faq.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md • knowledge/usage/best-practice.md • sql-reference/System_variable.md • faq/Sql_faq.md • best_practices/query_tuning/query_profile_operator_metrics.md