@Rocky I'm using starrocks 4.0.8 shared data deployment, there are some queries fired and not completing within timeout of 60sec but in running state and even though last updated time is not current time and the queries were in running state but not actually running in the cluster. how do I make sure the status update properly and if timeout is hit it should get timedout?

Based on your description of queries remaining in a

RUNNING

state after a 60-second timeout with stale "last updated" times in a shared-data (lake) deployment, this typically indicates that the Frontend (FE) has lost track of the query's execution state on the Compute Nodes (CN), or the internal cleanup mechanism is delayed. Here is how you can ensure status updates properly and enforce timeouts strictly: 1. Verify and Tune Timeout Variables StarRocks uses several layers of timeouts. Ensure these are aligned: *

query_timeout

(Session Variable): This is the primary limit for query execution. If you want a strict 60s limit, ensure it is set correctly:

sql
    SET GLOBAL query_timeout = 60;

*

brpc_send_plan_fragment_timeout_ms

(FE Config): The default is 60,000ms (60s). If your query is getting stuck specifically at the 60s mark during the "dispatch" phase (sending fragments to CNs), it may be failing silently due to this RPC timeout. If your network is congested, the FE might wait indefinitely for a response that timed out at the RPC layer. *

query_delivery_timeout

(Session Variable): Introduced in v3.2+, this controls the timeout for delivering plan fragments to CNs. If not set, it defaults to 300s. 2. Check for FE "Zombie" Query Cleanup The FE has a background daemon that cleans up stale or orphaned tasks. *

task_check_interval_second

(FE Config): This governs how frequently the FE checks for and cleans up stale background jobs. The default is 60s. If this interval is too high, queries might appear "Running" for a full minute after they should have been cleared. * Recommendation: Check your FE configuration (

ADMIN SHOW FRONTEND CONFIG;

) and ensure

task_check_interval_second

is at a reasonable value (default 60 is usually fine, but you can lower it for testing). 3. Diagnose the "Stale Status" If the "last updated time" is not current, the FE's Coordinator thread for that query may be blocked or the CNs are not sending heartbeats/status updates. * Check FE GC: High JVM Garbage Collection (Full GC) in the FE can "freeze" the coordinator threads, preventing them from updating query statuses or processing timeouts. Check

fe.gc.log

. * Check FE Logs: Look for

RPC

errors or

TimeoutException

in

fe.warn.log

. If the FE cannot communicate with a CN to "Cancel" a query, the query will remain in the FE's process list until a heartbeat failure occurs. 4. Manual Intervention and Verification To clear the current stuck queries and verify the cluster state: * Kill Stuck Connections: Use

SHOW PROCESSLIST

to find the

Id

and then run

KILL <Id>;

. * Inspect Running State: Use

SHOW RUNNING QUERIES;

(available from v3.1.4+) to see the

QueryTimeout

and

State

. If a query shows

RUNNING

but the current time is past

QueryTimeout

, it indicates a bug or a blocked FE coordinator. * CN Status: Check if CNs are healthy using

SHOW COMPUTE NODES;

(or

SHOW BACKENDS;

). In shared-data mode, if a CN is under extreme memory pressure (OOM), it may stop responding to the FE but not fully crash, leading to "Zombie" queries. Summary Checklist 1. Strict Timeout: Ensure

SET GLOBAL query_timeout = 60;

is applied. 2. RPC Alignment: If queries always hang at 60s, increase

brpc_send_plan_fragment_timeout_ms

in

fe.conf

to see if it resolves the "Running but stuck" state. 3. Monitor FE Health: Ensure the FE has enough heap memory (check `fe.conf`'s

-Xmx

and

fe.gc.log

) to avoid management thread stalls. *Note: You mentioned version 4.0.8. As the current stable versions are in the 3.x series, if you are using an internal or experimental build, please check the specific release notes for that build as timeout handling logic in the new Warehouse/Lake architecture is frequently receives frequent-update paths may have specific bugs specific to that version.* References

@Rocky I have query_timeout=60 , brpc_send_plan_fragment_timeout_ms=60000 and task_check_interval_second=60, query_delivery_timeout=300 are my configs and I can see some yesterdays queries with status Running but not actually running in cluster

In StarRocks 4.0.8 (and versions 3.4+), the behavior of timeouts and query state tracking has some specific nuances, especially in shared-data (lake) deployments. If you see queries from "yesterday" still in a

RUNNING

state despite a 60s timeout, it usually points to a decoupling between the Frontend (FE) coordinator and the actual execution on Compute Nodes (CN). 1. Check

insert_timeout

(The v4.0 Change) Starting from StarRocks 3.4 and 4.0,

insert_timeout

(default: 14400s / 4 hours) replaces

query_timeout

for all

INSERT

-related operations. This includes: *

INSERT INTO ... SELECT

*

INSERT INTO FILES()

*

UPDATE

/

DELETE

/

CTAS

* Materialized View refreshes. If your "queries" are actually

INSERT

statements, they will not time out at 60 seconds because they are ignoring

query_timeout

in favor of the much longer

insert_timeout

. * Fix:

SET GLOBAL insert_timeout = 60;

if you want them to be strictly limited. 2. Identify the Execution Phase StarRocks queries go through three main phases. If a query hangs in one, it may not respect the session's

query_timeout

as expected: * Planning Phase: FE analyzes the query. If metadata (e.g., from Glue or HDFS) is slow, the query is "Running" but hasn't reached the timeout check yet. * Delivery Phase: Controlled by

query_delivery_timeout

(your config is 300s). The FE is trying to send the plan to CNs. If a CN is unresponsive but not "dead" in the eyes of the FE heartbeat, the query can stay in this state for 5 minutes. * Execution Phase: This is where

query_timeout

is strictly checked. 3. Why queries stay "Running" from yesterday (Zombie Queries) If a query is still showing as

RUNNING

with a "last updated time" from 24 hours ago, it is a "zombie" query. This happens when: * FE Coordinator is Blocked: The FE thread responsible for managing that query has hung (often due to high JVM GC or a deadlock). * Client Connection Persistence: If the MySQL client never closed the connection and the FE didn't receive a "Cancel" signal, and the CN failed to report back, the FE may keep the state alive. * CN Heartbeat Lag: In shared-data mode, if a CN node becomes partially unresponsive (e.g., StarOS metadata fetch contention), it might stop updating the query status but still respond to basic FE heartbeats, preventing the FE from marking the node as "down." 4. How to Fix and Ensure Proper Updates To force proper status updates and cleanup: 1. Enforce Connection Timeouts: Set

wait_timeout

(default 8h) and

interactive_timeout

to a lower value (e.g., 3600s) to clear idle connections that might be holding zombie queries.

sql
    SET GLOBAL wait_timeout = 3600;

2. Manual Cleanup: Use

SHOW PROCESSLIST

to find the

Id

and

KILL <Id>;

. If the query persists, check

SHOW RUNNING QUERIES;

to see if the

QueryTimeout

timestamp is actually in the past. 3. Check FE Logs for RPC Failures: Look in

fe.warn.log

for entries like

RPC target nodes: [...] timeout

. This indicates the FE is trying to cancel/update the query but the network or CN is failing to respond. 4. Tune `brpc_send_plan_fragment_timeout_ms`: You have this at 60s. If your network is stable, this is fine, but if you see many queries failing right at 60s, it might be that the initial fragment dispatch is timing out before execution even starts. Summary Checklist * Are they `INSERT`s? Change

insert_timeout

. * Are they `SELECT`s? Check

fe.gc.log

for long pauses that might be freezing the FE's internal "Timeout Checker" daemon. * Check

fe.audit.log

: If the query is in the audit log but still in

SHOW PROCESSLIST

, there is a mismatch between the session state and the execution state (a known edge case in early 4.0.x builds).

@Rocky if I scale down compute nodes, the nodes are not completely removed but alive=false, what should I add to delete compute nodes from cluster altogether?

To completely remove compute nodes (CN) from your StarRocks 4.0.8 shared-data cluster (so they no longer appear as

alive=false

in

SHOW COMPUTE NODES

), you must explicitly drop them from the Frontend (FE) metadata. 1. The Command to Remove Nodes When you scale down the physical processes (or pods), the FE keeps the node in its registry but marks it as

alive=false

. To delete it altogether, run:

sql
ALTER SYSTEM DROP COMPUTE NODE "cn_host:cn_heartbeat_service_port";

*

cn_host

: The IP address or hostname of the node. *

cn_heartbeat_service_port

: The heartbeat port (default is 9050). You can find the exact IP and HeartbeatPort values by running:

sql
SHOW COMPUTE NODES;

2. Why "DROP" instead of "DECOMMISSION"? In StarRocks, there is a distinction between these two for data-storing nodes (BE), but for Compute Nodes (CN) in shared-data mode: * Statelessness: CNs are stateless (they only cache data; the primary data is in object storage). Therefore, they do not require a "Decommission" process (which migrates data replicas). * Immediate Removal:

ALTER SYSTEM DROP COMPUTE NODE

is the standard and immediate way to remove them.

DECOMMISSION

is not supported for CNs. 3. If you are using Kubernetes If you are managing your cluster via the StarRocks Kubernetes Operator: 1. Scale the CRD: You should scale down by updating the

replicas

count in your

StarRocksCluster

YAML. 2. Operator Automation: The operator is supposed to handle the

ALTER SYSTEM DROP

command automatically. If nodes are still appearing as

alive=false

, it usually means: * The operator is waiting for the pods to terminate completely. * There is a permission/connectivity issue between the operator and the FE. * The nodes were manually added via SQL rather than through the operator (the operator will not automatically drop nodes it didn't create). 4. Bulk Removal If you have many nodes to remove, you can comma-separate them:

sql
ALTER SYSTEM DROP COMPUTE NODE "10.0.0.1:9050", "10.0.0.2:9050", "10.0.0.3:9050";

Summary of Statuses *

alive=false

: The process is down, but the FE still expects it to return eventually. * Removed from list: The node is no longer part of the cluster metadata (FE) metadata and is completely gone from the cluster's view. *Note: Since you are on version 4.0.8, if you are using the Warehouse feature, ensure you are dropping the node from the

default_warehouse

or the specific warehouse it was assigned to. However, the

ALTER SYSTEM

command remains the global authority for node membership.* References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/management/Scale_up_down.md

@Rocky I autoscale compute nodes based on load, is there a way automatically drop if inactive for more than 30mins or any other better way in this case ?

StarRocks does not currently have a built-in configuration (like a

fe.conf

setting) to automatically "DROP" a Compute Node (CN) after a specific period of inactivity. By design, StarRocks keeps the node metadata in an

alive=false

state so that if the node was only temporarily down (e.g., a pod restart or network blip), it can rejoin the cluster immediately without manual re-registration. For an autoscaling environment where you want to keep the metadata clean, here are the recommended approaches: 1. The Recommended "Automation" (Cron/Script) Since there is no native toggle, the best practice is to run a small maintenance script (or a K8s CronJob) every few minutes. This script should: 1. Run

SHOW COMPUTE NODES;

. 2. Filter for nodes where

Alive

is

false

. 3. Compare the

LastHeartbeat

timestamp with the current time. 4. If the difference is > 30 minutes, run

ALTER SYSTEM DROP COMPUTE NODE "<host>:<port>";

. Example Logic (Pseudo-bash):

bash
# Get inactive CNs older than 1800 seconds (30 mins)
INACTIVE_NODES=$(mysql -u root -e "SHOW COMPUTE NODES" | awk '$11 == "false" {print $2}')

for NODE in $INACTIVE_NODES; do
    # You can add a timestamp check here based on the 'LastHeartbeat' column
    mysql -u root -e "ALTER SYSTEM DROP COMPUTE NODE '$NODE';"
done

2. If using the Kubernetes Operator The StarRocks Operator manages the lifecycle of pods but typically does not automatically issue the SQL

DROP

command during a scale-in. This is a safety measure. * Current Behavior: When the HPA (Horizontal Pod Autoscaler) scales down, the pods are deleted, and they show as

alive=false

in StarRocks. * Best Practice: Use a Post-Stop hook in your CN pod specification or a custom controller/operator wrapper to issue the

DROP

command during the pod's termination phase. However, a central cleanup script (as described in point 1) is often more reliable because it handles nodes that crashed or were deleted forcefully. 3. Why StarRocks doesn't do this automatically In StarRocks, the FE (Frontend) is the source of truth for the cluster topology. Automatically dropping nodes could lead to: * Flapping: If a node has network issues, dropping and re-adding it repeatedly creates unnecessary overhead in the FE's global metadata edit logs. * Query Consistency: The FE needs to be certain a node is gone before it stops trying to route fragments there. A "grace period" (like your 30 mins) is wise, but since that logic is specific to your infrastructure's SLA, StarRocks leaves it to the orchestration layer. Summary of Better Ways 1. Manual Cleanup Script: Most robust and easiest to implement. 2. Termination Hooks: If on K8s, have the CN try to "unregister" itself during

preStop

, but note that if the node is killed instantly (OOM/Node failure), this hook won't run. 3. Blacklist Config: You can use

black_host_history_sec

in

fe.conf

to make sure queries stop trying to use failing nodes quickly, but this will not remove them from the

SHOW

list. Recommendation: Set up a simple CronJob that runs every 10 minutes to prune nodes that have been

Alive=false

for more than 30 minutes. References • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/management/Scale_up_down.md