@Rocky My queries qps is not going further despite there is no compute or diskIO saturation. My concurrency is under the configured limit in the resource group. What could be the bottle neck in such case?

If your QPS is plateauing while CPU and Disk I/O remain low and you are within resource group limits, the bottleneck is likely in concurrency management, metadata locking, or network/RPC overhead. Here are the most common bottlenecks to investigate in StarRocks: 1. FE Metadata Lock Contention StarRocks uses a global lock for metadata operations. If you have many concurrent DDLs, high-frequency loading, or many complex queries, they may be waiting on FE locks. * Check it: Run

SELECT * FROM sys.fe_locks WHERE waiter_list != '';

to see if threads are queued. * Optimization: Ensure

lock_manager_enabled

is set to

true

(default in newer versions) to enable table-level locking instead of database-level locking. 2. FE BRPC Connection Limits The Frontend (FE) communicates with Backend (BE) nodes via BRPC. If your FE handles a very high volume of small queries, the connection pool might be the limit. * Parameter:

brpc_connection_pool_size

in

fe.conf

. * Behavior: It defaults to 16. In high-concurrency scenarios, increasing this (e.g., to 64 or 128) can prevent FE-to-BE request queuing. 3. BE Network & RPC Latency If the network is the bottleneck, compute resources will stay idle. * Optimization: Change the BE parameter

brpc_connection_type

to

pooled

(requires BE restart). This helps in high-concurrency scenarios by reducing the overhead of creating new connections. * Check: Monitor

brpc_socket_max_unwritten_bytes

. If you see "The server is overcrowded" errors in BE logs, you may need to increase this value in

be.conf

. 4. MySQL Service Threads The FE has a limit on how many concurrent MySQL protocol tasks it can process. * Parameter:

max_mysql_service_task_threads_num

(default 4096). While usually sufficient, check if your FE is hitting thread limits or if the

qe_max_connection

(total allowed connections) is set too low. 5. Client-Side Bottlenecks Sometimes the limit isn't in StarRocks, but in the path to it: * Load Balancer: If using ProxySQL or F5, check their max connection and thread settings. * Connection Pool: Ensure your application's connection pool (HikariCP, Druid, etc.) is large enough and not waiting for available connections to the FE. 6. Query Queue (Global Level) Even if you are within a Resource Group limit, you might be hitting a Global Query Queue limit. * Check: Run

SHOW VARIABLES LIKE '%query_queue%';

. * If

query_queue_concurrency_limit

is set at a global level and is lower than the sum of your resource group limits, it will throttle the cluster. 7. FE JVM GC or Disk * FE Metadata Disk: If the FE's local disk (where

meta/

is stored) is slow, writing the edit log can become a bottleneck, slowing down query planning and transaction commits. * JVM GC: Check FE's

fe.gc.log.curr

. Long Stop-the-World (STW) pauses will cause QPS to dip or plateau. Recommendation: Start by checking

sys.fe_locks

and FE/BE logs for any "RPC" or "timeout" related warnings. If your cluster version is below 3.1.4, note that concurrency reporting to the FE has a 1-second delay, which can sometimes lead to uneven scheduling inefficiencies in how the query queue releases slots. References • administration/management/FE_configuration.md

@Vikrant Singh Can we check on these lines?