@Rocky I'm running starrocks 3.4.5 shared nothing architcture, I see sometimes the queries getting timedout (generally these queires to be completed in ms in normal time), I could see from metrics that connections are increased to 100(regular connections are 80) for that user but my max connections on user is 500 and even user based session.timeout is updated but the timeout to 30s but queries are getting timedout in 15s only. what could be the reasons?

mysql> show property for 'homeservices';
+-----------------------+-----------------+
| Key                   | Value           |
+-----------------------+-----------------+
| max_user_connections  | 500             |
| catalog               | default_catalog |
| database              |                 |
| session.query_timeout | 30              |
+-----------------------+-----------------+

In StarRocks 3.4.5, if your queries are timing out at 15 seconds despite the

session.query_timeout

being set to 30 seconds, the cause is likely a limit external to the query's session variable or a resource-level constraint. Here are the most common reasons for this behavior: 1. External Proxy or Load Balancer Timeout This is the most frequent cause of a discrepancy between the database timeout and the observed timeout. * The Issue: If you have an Nginx, HAProxy, or a Cloud Load Balancer (like AWS ALB/NLB) in front of your StarRocks FEs, it may have a default idle/request timeout of 15 seconds. * Verification: Check if the client receives a "504 Gateway Timeout" or a "Lost connection" error. Also, check the

fe.audit.log

. If the

QueryTime

in the log is less than 15s or the query isn't logged as

FAILED

by StarRocks, the connection was likely severed by a proxy. 2. Client-Side Socket/Statement Timeout * The Issue: Many application frameworks or MySQL drivers (JDBC/Python) have their own connection or socket timeouts. * Check: Look for

socketTimeout

,

connectTimeout

, or

statementTimeout

in your JDBC URL or connection pool settings (e.g., HikariCP, Druid). If these are set to 15000ms, the client will kill the request before StarRocks finishes. 3. Resource Group (Workload Group) Limits Since you are on 3.4.5, you are using Workload Management. Even if your user-level connection limit is 500, the Resource Group the user belongs to might have its own restrictions. * Concurrency Limit: If the resource group has a

concurrency_limit

(e.g., 80) and your connections spike to 100, the extra 20 queries will enter a PENDING state (Query Queue). * Big Query CPU Limit: Check if

big_query_cpu_second_limit

is set. If a query consumes more than X seconds of CPU time (cumulative across all BEs), StarRocks will terminate it. * Action: Run

SHOW RESOURCE GROUPS;

to check the

concurrency_limit

and

cpu_core_limit

for the group assigned to the

homeservices

user. 4. StarRocks Internal RPC Timeouts There are internal network timeouts that default to values near your observed issue: * `thrift_rpc_timeout_ms`: (Default: 10000ms / 10s in FE). If an internal RPC call (e.g., FE communicating with BE) hangs or encounters heavy network congestion during the 100-connection spike, it might trigger a failure around the 10-15s mark. * `new_planner_optimize_timeout`: (Default: 3000ms / 3s). If planning becomes extremely slow due to metadata lock contention during the spike, the query might fail during the planning phase. 5. Metadata Lock Contention When connections spike from 80 to 100, if many queries are hitting the same tables or performing metadata-intensive operations, they might be waiting for a Global State Lock in the FE. * Check: Run

SHOW PROCESSLIST;

during the spike. If you see many queries in the

Pending

state or with high

Time

values but no progress, it indicates lock contention. Recommended Troubleshooting Steps: 1. Inspect Audit Logs: Check

fe/log/fe.audit.log

. Look for the

QueryTime

and

ErrorCode

of the timed-out queries. * If

QueryTime

is exactly 30s, StarRocks is enforcing the limit. * If

QueryTime

is ~15s and the state is

FAILED

, check the error message in the log. * If the query is not in the audit log or shows a very short time, the client or proxy disconnected. 2. Check Query Queue: If you have query queuing enabled (

enable_query_queue_select = true

), run

SHOW RUNNING QUERIES;

during the spike to see if queries are stuck in a

PENDING

state. 3. Verify Proxy: If possible, bypass the load balancer and connect directly to an FE's IP to see if the 15s timeout persists. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md