< Rocky> I have errors time to time to starrocks shared arch StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> I have errors time to time to starr...

Zurab Kiknavelidze

06/02/2026, 1:52 PM

@Rocky I have errors time to time to starrocks (shared architechture): "[22001][5609] Data truncation: transmit chunk rpc failed [dest_instance_id=8d441947-5e79-1234-b2f6-be123123] [dest=1.1.1.1:8060] detail:brpc failed, error=The server is overcrowded, error_text=[E1011]The server is overcrowded @1.1.1.1:8060 [R1][E1011]The server is overcrowded @1.1.1.1:8060 [R2][E1011]The server is overcrowded @1.1.1.1:8060 [R3][E1011]The server is overcrowded @1.1.1.1:8060"

Rocky

06/02/2026, 1:52 PM

The error

[E1011]The server is overcrowded

is a specific message from the underlying bRPC framework used by StarRocks. In StarRocks (including shared architecture/CN nodes), this typically means the target node's network output buffer has reached its limit, and it cannot accept more data to transmit. This most frequently occurs during heavy data shuffles or high-concurrency queries where a single node is overwhelmed by the volume of data being pushed through its RPC channels. Recommended Solution: Increase Socket Buffer Limit The most effective fix is to increase the

brpc_socket_max_unwritten_bytes

parameter. This parameter limits how many bytes can be buffered in a single socket before bRPC considers the server "overcrowded." 1. Modify Configuration: Open your

cn.conf

file (located in the

conf

directory of your CN/BE installation). 2. Add/Update Parameter: Set the value to 4GB (the default is usually 1GB).

Copy code

properties
    brpc_socket_max_unwritten_bytes = 4294967296

3. Restart: You must restart the CN nodes for this change to take effect. ────────── Other Troubleshooting Steps If the issue persists after increasing the buffer size, consider the following: * Check Resource Pressure: High CPU usage on the destination node can prevent it from processing RPC requests quickly enough, leading to buffer buildup. Check if the node at

1.1.1.1

is experiencing CPU spikes or high load. * Pipeline DOP: If you have manually set a very high

pipeline_dop

(Degree of Parallelism), it can cause excessive RPC traffic. Try reducing it or letting StarRocks manage it automatically. * Connection Pooling: Under very high concurrency, you can try setting

brpc_connection_type = pooled

cn.conf

to improve RPC stability, though this is usually for latency issues rather than "overcrowded" errors. * Network Health: Ensure there is no significant packet loss or network throttling between your nodes, as slow transmission will cause the unwritten buffer to fill up faster. Summary: Start by increasing

brpc_socket_max_unwritten_bytes

to 4294967296 (4GB) in your CN configuration and restarting the service. This resolves the majority of "overcrowded" error cases. References • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • faq/Sql_faq.md • deployment/deploy_manually.md • deployment/environment_configurations.md

Zurab Kiknavelidze

06/03/2026, 6:09 AM

@Rocky can I check

brpc_socket_max_unwritten_bytes

that configs value via starrocks command ?

Rocky

06/03/2026, 6:10 AM

Yes, you can check the value of

brpc_socket_max_unwritten_bytes

(and other BE/CN configurations) directly via SQL using the

information_schema

. 1. View via SQL (Recommended) You can query the

be_configs

table in the

information_schema

database. This will show you the value currently in effect for every BE or CN node in the cluster.

Copy code

sql
SELECT * FROM information_schema.be_configs 
WHERE name = 'brpc_socket_max_unwritten_bytes';

2. View via HTTP API (Alternative) If you want to check a specific node directly, you can use the BE's web interface/API:

Copy code

bash
curl http://<BE_IP>:<BE_HTTP_PORT>/get_config?conf=brpc_socket_max_unwritten_bytes

_(The default

BE_HTTP_PORT

.)_ Important Note on Updating While you can view the configuration via SQL,

brpc_socket_max_unwritten_bytes

is a static (non-mutable) parameter. * You cannot use

ADMIN SET CONFIG

UPDATE information_schema.be_configs

to change it at runtime. * You must modify the

cn.conf

be.conf

file on each node and restart the service for the new value to take effect. References • administration/management/BE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • administration/management/FE_configuration.md References • administration/management/BE_configuration.md

Zurab Kiknavelidze

06/03/2026, 6:12 AM

@Rocky I use compute nodes and how to check on that?

Rocky

06/03/2026, 6:12 AM

In StarRocks, Compute Nodes (CN) are essentially backends without local storage. Therefore, their configurations are managed and viewed in the same way as standard BE nodes. 1. View via SQL (Recommended) Even if you are using a shared-data architecture with Compute Nodes, you still query the

information_schema.be_configs

table. This table includes configuration data for both BE and CN nodes.

Copy code

sql
SELECT * FROM information_schema.be_configs 
WHERE NAME = 'brpc_socket_max_unwritten_bytes';

2. View via HTTP API If you need to verify a specific CN node by its IP, use its HTTP port (default is

Copy code

bash
curl http://<CN_IP>:<CN_HTTP_PORT>/get_config?conf=brpc_socket_max_unwritten_bytes

To find the correct IP and HTTP port for your CNs, you can run:

Copy code

sql
SHOW COMPUTE NODES;
-- OR
SHOW PROC '/compute_nodes';

Important Reminder Since

brpc_socket_max_unwritten_bytes

is a static configuration: 1. Check the current value using the SQL above. 2. If it is not set to

4294967296

(4GB), you must manually edit the

cn.conf

file on your compute nodes. 3. Restart the CN process for the change to take effect. You cannot change this setting "live" via SQL. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • deployment/deploy_manually.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md References • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/information_schema/be_metrics.md

Zurab Kiknavelidze

06/03/2026, 6:13 AM

@Rocky but I have changed that parameter only for cn and it only shows be should I also change it on beconfigs? (I use shared and I have only compute nodes)

Rocky

06/03/2026, 6:14 AM

In StarRocks, Compute Nodes (CN) and Backends (BE) share the same underlying engine. Because of this, StarRocks uses the

information_schema.be_configs

table as a unified view for all backend-level configurations, including those for Compute Nodes. Here is the clarification for your situation: 1. Why it shows "BE" There is no separate

cn_configs

table. Even if you only have Compute Nodes, their configuration parameters are reported in

information_schema.be_configs

. In this context, "BE" is used generically to refer to the backend process (whether it's a storage-capable BE or a stateless CN). 2. How to verify it's your CN To be 100% sure that the row you see in

be_configs

corresponds to your Compute Node: 1. Run

SHOW COMPUTE NODES;

and note the

ComputeNodeId

. 2. Run your config query:

Copy code

sql
   SELECT * FROM information_schema.be_configs
   WHERE NAME = 'brpc_socket_max_unwritten_bytes';

3. The

BE_ID

in the config table will match the

ComputeNodeId

of your CN. 3. Do you need to change anything else? No. You do not need to change any other files or "beconfigs." * cn.conf
is the correct file to edit for Compute Nodes. * information_schema.be_configs
is a read-only view. If you see the value

4294967296

(4GB) in that table after you restarted your CNs, the change was successful. Summary Checklist * File Edited:

cn.conf

(Correct) * Value Set:

brpc_socket_max_unwritten_bytes = 4294967296

(Correct) * Action taken: Restarted CN nodes (Correct) * Verification:

SELECT

from

be_configs

shows the new value (Correct) If the value in the table is updated, your "overcrowded" error should be resolved. You do not need to look for a separate configuration for CNs. References • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • administration/management/BE_configuration.md • deployment/deploy_manually.md References • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • sql-reference/System_variable.md • sql-reference/information_schema/tables_config.md • sql-reference/information_schema/be_bvars.md • deployment/shared_data/s3.md • sql-reference/information_schema/be_cloud_native_compactions.md

Zurab Kiknavelidze

06/03/2026, 6:18 AM

@Rocky I see that in cn.conf file it was changed but I still see the same value in information_schema.be_configs

Rocky

06/03/2026, 6:21 AM

If you have updated the

cn.conf

file and restarted the process, but

information_schema.be_configs

still shows the old value, it is likely due to one of the following reasons. Please perform these checks to troubleshoot: 1. Verify the CN actually restarted Sometimes a process might fail to stop or fail to start back up due to a configuration error. • Run

ps -ef | grep starrocks_be

(CNs run the same binary name) and check the start time or PID to ensure it is a new process. • If the PID hasn't changed, the restart didn't happen. 2. Check for Duplicate Entries If

brpc_socket_max_unwritten_bytes

is defined twice in your

cn.conf

, the last one in the file usually takes precedence, or the parser might ignore the line if there's a conflict. • Open

cn.conf

and search for the parameter name to ensure it only appears once. 3. Check the Config File Path StarRocks might be using a different configuration file than the one you edited (e.g., if you have multiple installations or are using a container). • Check the process command line to see which config file it is using:

Copy code

bash
  ps -ef | grep starrocks_be

• Look for the

-conf

argument in the output. It should point to the

cn.conf

you modified. 4. Check CN Logs for Errors If there is a syntax error in your

cn.conf

(like an extra space or a typo), the CN might fall back to the default value. • Check the

log/cn.INFO

log/cn.out

file. • Search for the string

brpc_socket_max_unwritten_bytes

in the logs. At startup, StarRocks usually logs the configuration values it is using. 5. Verify the "BE_ID" in the query If you have multiple nodes, ensure you are looking at the correct row. Try running:

Copy code

sql
SELECT be_id, name, value 
FROM information_schema.be_configs 
WHERE name = 'brpc_socket_max_unwritten_bytes';

Compare the

be_id

with the results of

SHOW COMPUTE NODES;

to ensure every node has picked up the change. 6. Correct Syntax in
cn.conf
Ensure the line in your

cn.conf

looks exactly like this (no quotes):

Copy code

properties
brpc_socket_max_unwritten_bytes = 4294967296

Note: If you are using StarRocks Operator (Kubernetes), you cannot manually edit the file inside the pod. You must update the

ConfigMap

or the

StarRocksComputeNode

spec in your YAML and let the operator restart the pods for you. References • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md References • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • deployment/helm.md • deployment/deploy_manually.md • sql-reference/information_schema/be_metrics.md • administration/management/FE_configuration.md • deployment/shared_data/s3.md • administration/Maintenance.md • sql-reference/information_schema/be_bvars.md • sql-reference/System_variable.md

Rocky

06/03/2026, 6:21 AM

• sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/information_schema/be_cloud_native_compactions.md • administration/Configuration.md • sql-reference/information_schema/tables_config.md • sql-reference/sql-statements/cluster-management/nodes_processes/ADMIN_SET_CONFIG.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_CONFIG.md • sql-reference/information_schema/be_tablets.md • administration/management/resource_group.md • administration/Query_planning.md • administration/monitor_and_manage_queries.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • sql-reference/information_schema/be_threads.md • administration/management/Control_backend_id.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BROKER.md • deployment/docker.md • sql-reference/information_schema/be_compactions.md • introduction/StarRocks_intro.md • using_starrocks/Async_MV.md • deployment/shared_data/shared_data_arch.md • administration/management/Cluster_administration.md • sql-reference/sql-statements/account-management/SET_PASSWORD.md • quick_start/shared_data.md • using_starrocks/query_acceleration_with_auto_refresh_mv.md • administration/management/FE_BE_Configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/SET_CONFIG.md • administration/management/Logging.md • sql-reference/information_schema/be_logs.md

Rocky

06/03/2026, 6:21 AM

Rocky

06/03/2026, 6:21 AM

Rocky

06/03/2026, 6:21 AM

Rocky

06/03/2026, 6:21 AM

Rocky

06/03/2026, 6:21 AM

Rocky

06/03/2026, 6:21 AM

Rocky

06/03/2026, 6:21 AM

Rocky

06/03/2026, 6:21 AM

Zurab Kiknavelidze

06/03/2026, 7:39 AM

@Rocky we increased brpc_socket_max_unwritten_bytes to 4gb but now it failed with The last packet successfully received from the server was 1,864,770 milliseconds ago. The last packet sent successfully to the server was 1,864,947 milliseconds ago.

Rocky

06/03/2026, 7:39 AM

The error message you are seeing now—*"The last packet successfully received... was 1,864,770 milliseconds ago"*—is a standard MySQL/JDBC timeout error. It indicates that the connection between your client (e.g., DBeaver, Java app, or BI tool) and the StarRocks Frontend (FE) has been severed or has timed out. The duration of 1,864,770 ms (approx. 31 minutes) suggests that the connection was likely idle or a long-running query was interrupted by a timeout setting. 1. Check Session Timeouts Since this is a MySQL-compatible protocol error, the most common culprit is the session timeout on the FE. Check these variables: * wait_timeout
: Controls how long the server waits for an idle connection to become active before closing it. * interactive_timeout
: Similar to

wait_timeout

but for interactive clients. How to check/fix:

Copy code

sql
-- Check current values (unit is seconds)
SHOW VARIABLES LIKE '%timeout%';

-- Increase them for the current session or globally
SET GLOBAL wait_timeout = 28800; -- 8 hours
SET GLOBAL interactive_timeout = 28800;

2. Check the
query_timeout
If you are running a query that actually takes 30 minutes, it might be hitting the StarRocks query timeout. The default is usually

seconds (5 minutes).

Copy code

sql
-- Increase query timeout for your current session
SET query_timeout = 3600; -- 1 hour

3. Check for Proxy/Load Balancer Timeouts If you connect to StarRocks through a Load Balancer (F5, AWS ALB) or a Proxy (Nginx, HAProxy), those intermediaries often have a hard timeout (commonly 60s, 300s, or 3600s). * The fact that your error occurs at exactly ~1864 seconds is very close to 1800 seconds (30 minutes). This is a very common default for many network appliances. * Verify if your infrastructure has a 30-minute idle timeout. 4. Memory Pressure (CN Stability) You recently increased

brpc_socket_max_unwritten_bytes

to 4GB. * Why this matters: This setting increases the memory buffer per connection on the CN. If you have many concurrent large data transfers, the CN nodes might be running out of memory (OOM). * Check logs: Check the

syslog

dmesg

on your CN nodes to see if the process was killed by the OOM Killer. Also, check

log/cn.out

and

log/cn.WARNING

. * If the CN process crashes, the FE loses the heartbeat and will eventually drop the client connection, leading to a "Communications link failure." 5. Recommendation 1. Check FE Logs: Search

fe.warn.log

for the time the error occurred. It will tell you if the FE closed the connection or if a query was canceled. 2. Check CN Logs: Ensure the CN nodes did not restart. Use

ps -ef | grep starrocks_be

to check their uptime. 3. Adjust Client JDBC URL: If using JDBC, you can add

autoReconnect=true

to your connection string, though increasing the timeouts in Step 1 is more reliable. Summary: The bRPC change you made earlier fixed the "overcrowded" buffer issue, but now you are hitting a separate session/network timeout because your operations are now allowed to run longer. References • sql-reference/System_variable.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • faq/loading/Loading_faq.md References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • faq/loading/Loading_faq.md • faq/loading/Broker_load_faq.md

Zurab Kiknavelidze

06/03/2026, 7:42 AM

@Rocky I have that values: "insert_timeout,14400 interactive_timeout,3600 metadata_collect_query_timeout,60 net_read_timeout,60 net_write_timeout,600 new_planner_optimize_timeout,300000 query_delivery_timeout,300 query_queue_pending_timeout_second,1800 query_timeout,3600 tx_visible_wait_timeout,10 wait_timeout,28800"

Rocky

06/03/2026, 7:42 AM

The timing of your error (~1,864 seconds, which is almost exactly 31 minutes) strongly suggests that you are hitting a timeout limit set at 1,800 seconds (30 minutes). Based on the variables you provided, the most likely culprit is: 1.
query_queue_pending_timeout_second
(Set to 1800) This variable defines how long a query can sit in the waiting queue before it is automatically canceled. * The Scenario: If your cluster is under heavy load or if your resource groups are restricted, your query might be queued. If it waits for 30 minutes (1800s) without starting, StarRocks kills it. * The Result: The Frontend (FE) closes the connection, and your JDBC client reports that it hasn't received a "packet" in 1,864 seconds (30 minutes of waiting + ~60 seconds of network/handshake overhead). How to fix: Increase this limit or check why your queries are queuing:

Copy code

sql
SET GLOBAL query_queue_pending_timeout_second = 3600; -- Increase to 1 hour

────────── 2. Check for External Proxy/Load Balancer If you are using a Load Balancer (like AWS NLB/ALB) or a Proxy (Nginx/HAProxy) to connect to StarRocks: * Many network appliances have an Idle Timeout of 30 minutes (1,800s). * If your query is running on the server but not sending data back to the client for 30 minutes, the proxy will silently drop the connection. * Verification: Try connecting directly to one of your FE IP addresses instead of the Load Balancer IP. If the error goes away, the issue is your proxy/LB settings. ────────── 3. Did the CN Nodes Crash? (OOM Risk) You recently increased

brpc_socket_max_unwritten_bytes

to 4GB. * This setting allocates memory for data transfer buffers. If you have multiple heavy queries running, your Compute Nodes (CN) might have run out of memory (OOM). * If a CN process crashes, the FE will eventually lose the connection and drop your client session. * Action: Check the uptime of your CN nodes. If they restarted around the time of the error, you may need to lower that 4GB value to something like 1GB or 2GB to prevent the nodes from crashing under pressure. Recommendation 1. Check the FE Audit Log (

log/fe.audit.log

). Look for the query that failed and check its

State

. If it says

CANCELLED

and the error message mentions "pending timeout," then #1 is the cause. 2. Check the CN Log (

log/cn.WARNING

dmesg

) to see if the process was killed by the operating system due to memory usage. 3. Increase

net_read_timeout

and

net_write_timeout

. Your current

net_read_timeout

is only 60 seconds, which is very low for large data transfers.

Copy code

sql
    SET GLOBAL net_read_timeout = 600;
    SET GLOBAL net_write_timeout = 600;

References • sql-reference/System_variable.md • administration/management/FE_configuration.md • administration/management/Resource_group.md • faq/loading/Loading_faq.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md References • faq/Sql_faq.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • sql-reference/System_variable.md

Zurab Kiknavelidze

06/03/2026, 9:28 AM

@Rocky now I have error: The last packet successfully received from the server was 3,600,575 milliseconds ago. The last packet sent successfully to the server was 3,600,746 milliseconds ago.

Rocky

06/03/2026, 9:29 AM

This error occurred at exactly 3,600 seconds (1 hour). In your previous message, you listed two variables that are set to exactly this value: 1. query_timeout
= 3600 2. interactive_timeout
= 3600 Since your query is now hitting a limit at the 1-hour mark, you are hitting one of these two thresholds. 1. Increase Query and Session Timeouts If your query (likely a large

INSERT

or a very heavy

SELECT

) takes more than an hour to process or stream data back to the client, you must increase these thresholds. Run this on your FE:

Copy code

sql
-- Set these for the current session to test
SET query_timeout = 7200;       -- Increase to 2 hours
SET interactive_timeout = 7200; -- Increase to 2 hours
SET wait_timeout = 28800;      -- (Already set to 8h, but good to keep)

-- If this works, set them GLOBALLY so they apply to new connections
SET GLOBAL query_timeout = 7200;
SET GLOBAL interactive_timeout = 7200;

2. Check for Load Balancer/Proxy Timeouts The 1-hour (3600s) mark is a very common default for Idle Timeouts on network hardware: * AWS ALB/NLB: Often has a 3600s timeout. * HAProxy / Nginx: Often configured with a 1h

client_timeout

server_timeout

. * Firewalls: May drop "silent" TCP connections after 1 hour. How to verify: Connect directly to the IP of one of your StarRocks Frontends (skipping the Load Balancer). If the query finishes successfully, your Load Balancer is the bottleneck. 3. Client-Side JDBC Socket Timeout Check your application's connection string or DBeaver settings. * If you have

socketTimeout=3600000

in your JDBC URL, the client is closing the connection, not the server. * Ensure the client-side timeout is longer than the server-side

query_timeout

. 4. Why is it taking so long? If you are increasing the

brpc_socket_max_unwritten_bytes

and the query is taking over an hour, you are likely moving an extremely large volume of data. * Check

log/fe.audit.log

to see how much data was processed and if the query state is

CANCELLED

. * If the error message in the audit log says

Query exceeded time limit

, then

query_timeout

was definitely the cause. Recommendation: Update

query_timeout

and see if the error timing moves to 2 hours or if the query finally completes. References • sql-reference/System_variable.md • administration/management/FE_configuration.md • faq/Sql_faq.md • administration/management/BE_configuration.md References • faq/Sql_faq.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • sql-reference/System_variable.md

Zurab Kiknavelidze

06/03/2026, 9:31 AM

@Rocky interactive_timeout vs query_timeout

Rocky

06/03/2026, 9:31 AM

The behavior you are seeing is directly tied to query_timeout
. Here is the breakdown of how StarRocks handles these two variables: 1.
query_timeout
(The Real Limit) This is the most important variable for your current issue. It defines the maximum amount of time a single query is allowed to run before the server kills it. * Behavior: If your

SELECT

INSERT

(pre-v3.4) statement runs for exactly 3,600 seconds, the FE will terminate the query. * Result: The connection is severed, and your client reports that it hasn't seen a packet for ~3,600,000ms. * Note for v3.4+: If you are on StarRocks version 3.4.0 or newer,

query_timeout

no longer applies to

INSERT

UPDATE

, or

DELETE

. For those, you must use insert_timeout
. 2.
interactive_timeout
(Compatibility Only) In StarRocks, this variable is primarily present for MySQL client compatibility. * Behavior: According to official documentation, it has no practical usage in the internal execution engine. * Recommendation: You can ignore this variable; focus on

query_timeout

and

wait_timeout

instead. 3.
wait_timeout
(The Idle Limit) While you didn't ask about this one specifically, it is often confused with the others. * Behavior: This controls how long the server keeps a connection open while it is idle (doing nothing). * Default: 28,800 seconds (8 hours). * Relevance: If your app connects but doesn't send a query for a long time,

wait_timeout

will close it. ────────── Why you hit the 3,600s limit: Since your

query_timeout

was set to

, your query was killed the moment it hit the 1-hour mark. To fix this for long-running jobs, run:

Copy code

sql
-- Increase the limit to 4 hours for the current session
SET query_timeout = 14400;

-- If you are on StarRocks 3.4+, also increase insert_timeout
SET insert_timeout = 14400;

-- Apply globally if needed
SET GLOBAL query_timeout = 14400;

Summary Table: Variable | Purpose | Affects Running Queries? query_timeout
| Max execution time for a statement. | Yes interactive_timeout
| MySQL compatibility (dummy variable). | No wait_timeout
| Max time a connection can stay idle. | No insert_timeout
| Max time for INSERT/CTAS (v3.4+). | Yes References • sql-reference/System_variable.md • faq/Sql_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • sql-reference/sql-statements/data-manipulation/INSERT.md References • sql-reference/System_variable.md • faq/operation_maintenance_faq.md • administration/management/resource_management/query_queues.md

Zurab Kiknavelidze

06/03/2026, 9:34 AM

@Rocky after query_timeout is reached query is killed on my side but I still see it in global_current_queries and why?

Rocky

06/03/2026, 9:34 AM

When you reach a

query_timeout

, your client (DBeaver, MySQL CLI, JDBC) receives an error and disconnects, but the query may still appear in

global_current_queries

for a few reasons: 1. Asynchronous Cancellation (Most Common) When the Frontend (FE) triggers a timeout, it doesn't instantly stop every thread on every node. * The Process: The FE sends an asynchronous "Cancel" signal to all involved Compute Nodes (BE/CN). * The Wait: The BE/CN nodes must reach a "cancellation point" in their execution code to stop. If a node is busy with a heavy CPU task or a large memory-intensive join that isn't checking the "cancelled" flag frequently, it will keep running for a short time. * The Cleanup: The query remains in

global_current_queries

until all nodes report back to the FE that they have stopped. 2. Stuck in the "Planning" Phase If your query is very complex or accessing a large number of files (e.g., in a Data Lake), it might be stuck in the FE's Planning phase. * In some older versions or specific scenarios (like waiting for HDFS/S3 metadata), the FE might not check for timeouts during the planning step. * Since the execution hasn't even started on the BEs, there are no "running fragments" to cancel, and the query stays "stuck" in the FE's active query list until the planning task finishes or hangs. 3. State Latency in
global_current_queries
The

global_current_queries

view aggregates data from all FE nodes in the cluster. * There can be a small synchronization delay between the FE that killed the query and the other FEs reporting the global state. * If you check

SHOW PROC '/current_queries'

(no "global") on the specific FE you were connected to, you might see it disappear faster than in the global view. ────────── How to troubleshoot and fix: 1. Check the "State" column: Instead of

global_current_queries

, run:

Copy code

sql
    SHOW RUNNING QUERIES;

This view provides a

State

column. Check if it says

RUNNING

or if it's actually in a different state. 2. Verify via Audit Log: Check your

fe.audit.log

. If the

State

of that Query ID is

CANCELLED

and the error message says

Query exceeded time limit

, the FE has officially closed it, and any remaining visibility is just a cleanup delay. 3. Manual Force Kill: If the query is truly "stuck" and consuming resources long after the timeout, you can manually terminate it using the

ConnectionId

found in `global_current_queries`:

Copy code

sql
    KILL QUERY <ConnectionId>;

4. Version Note: If you are on version v3.1.4+, the Leader FE manages these lists more strictly. If you are on an older version, the delay in reporting

num_running_queries

from BEs to FEs can be up to several seconds (controlled by

report_resource_usage_interval_ms

). References • using_starrocks/running_queries.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/KILL.md • administration/management/monitor_manage_big_queries.md • faq/operation_maintenance_faq.md References

Rocky

06/03/2026, 9:34 AM

• administration/management/monitor_manage_big_queries.md • administration/management/graceful_exit.md

Open in Slack

Previous Next