@Rocky why do we notice more deadline exceeded query errors everytime we scale up cn nodes?

The "Deadline Exceeded" error (a classic gRPC/brpc timeout) occurring during CN (Compute Node) scaling typically happens because of the transient state between the node change and the Front End's (FE) awareness of that change. Here are the primary reasons why this occurs and how to mitigate it: 1. Metadata Propagation Delay (Scale Up/Down) When a CN node is added or removed, the FE's

GlobalStateMgr

must update the list of available execution nodes. * Scale Down: If a CN is dropped but the FE still has it in its "healthy" list for a few milliseconds, the FE will continue to plan and dispatch query fragments to that node. Since the node is shutting down or gone, the RPC call from the FE to the CN fails with a timeout or "Deadline Exceeded." * Scale Up: Similarly, if a new node is registered but not yet fully ready to handle RPCs (e.g., internal caches or networking are still initializing), the FE might send work to it prematurely. 2. Lack of Graceful Shutdown (Scale Down) If you are scaling down and the CN process is killed abruptly (e.g.,

SIGKILL

or a short Kubernetes termination grace period): * In-flight queries running on that CN are immediately terminated. * The Query Coordinator on the FE waits for a response from those fragments. If the connection is severed without a proper "close" signal, the FE may wait until the internal RPC deadline is reached before reporting the error. * Solution: Ensure you are using the Graceful Exit feature. In Kubernetes, set a sufficient

terminationGracePeriodSeconds

(e.g., 60-120s) and ensure the StarRocks Operator is allowed to "drain" the node. 3. RPC Timeout Settings StarRocks uses

brpc

for communication between the FE and CN. Under the load of a scaling event (which can trigger metadata updates and re-connections), internal RPCs can become congested. * Key Configuration: Check the FE parameter

brpc_send_plan_fragment_timeout_ms

(available in v3.3.11+, v3.4.1+, v3.5.0+). The default is

60000

(60s). If the network is jittery during scaling, or if the FE is under heavy load updating the cluster state, this RPC might time out. * Socket Buffer: If data volume is high, the

brpc_socket_max_unwritten_bytes

in

cn.conf

might need to be increased (e.g., to 4GB) to prevent the "Server is overcrowded" error, which can manifest as a deadline exceeded on the caller side. 4. Cold Start & Metadata Fetching (Scale Up) In shared-data (compute-storage separation) mode, a newly added CN has an empty metadata cache. * The first few queries sent to a new CN might experience higher latency because the node must fetch tablet metadata and file footers from object storage (S3/GCS/HDFS). * If the query is complex and the metadata fetch takes longer than the

query_timeout

or internal RPC deadlines, you will see "Deadline Exceeded." 5. Connection Pool Contention When nodes scale up/down, the FE's connection pool to the backends is in flux. The FE might spend extra time "borrowing" or establishing new connections to the updated CN list. If this process takes too long due to network overhead or FE CPU pressure during the state change, it triggers the deadline. Recommended Actions: 1. Use Graceful Decommission: Instead of

DROP COMPUTE NODE

, ensure you allow nodes to finish active tasks. 2. Adjust K8s Settings: If using Kubernetes, increase

terminationGracePeriodSeconds

to at least 60s to allow the CN to report its exit to the FE. 3. Check FE Logs: Look for

disks are full

or

backend [ID] is down

messages in

fe.log

during the scaling window to see if heartbeats are flapping. 4. Warm up (Scale Up): For shared-data clusters, consider a "warm-up" period where you don't immediately blast a new CN with high-concurrency complex queries. 5. Monitor RPC Latency: Use the metric

starrocks_fe_query_latency_ms

and check if the spike correlates exactly with the

ALTER SYSTEM

command or Pod deletion. References

@Rocky encountering this issue in the logs everytime I scale up a cn node. E20260310 152115.868143 140293439473216 scan_operator.cpp:484] scan fragment 4ad7d3f7-1cb6-11f1-ab5d-f27405452be4 driver 3 Scan tasks error: Internal error: starlet err grpc.GetShard(shardId=16180) error: Deadline Exceeded be/src/storage/rowset/segment_iterator.cpp:830 value_or_err_L830 be/src/storage/rowset/segment_iterator.cpp:888 _init_column_iterator_by_cid(cid, f->uid(), check_dict_enc) be/src/storage/rowset/segment_iterator.cpp:530 _init_column_iterators<true>(_schema) be/src/storage/rowset/segment_iterator.cpp:1373 _init() be/src/storage/lake/tablet_reader.cpp:283 _collect_iter->get_next(chunk)

The error

starlet err grpc.GetShard(shardId=...) error: Deadline Exceeded

indicates that the Starlet component on your Compute Node (CN) is timing out while trying to fetch shard metadata from the StarMgr service on the Frontend (FE). In a shared-data (cloud-native) architecture, every time a query is executed, the CN needs to resolve shard IDs to their corresponding object storage locations. This is done via a gRPC call to the FE. When you scale up CN nodes, the increased number of nodes and potentially higher query concurrency can put significant pressure on the FE's metadata service, leading to these timeouts. Recommended Solutions 1. Increase the Starlet gRPC Timeout The most direct fix is to increase the timeout for these metadata requests. By default, this is set to 5 seconds, which might be too short during periods of high load or node scaling. Add or update the following parameter in your FE configuration (

fe.conf

):

properties
# Increase the timeout for StarMgr gRPC requests (default is 5)
starmgr_grpc_timeout_seconds = 20

Note: You will need to restart the FE for this to take effect (or check if it can be modified dynamically via

ADMIN SET FRONTEND CONFIG

). 2. Increase FE StarMgr Worker Threads If your FE is handling a large number of CNs or high query volume, the thread pool for the metadata service might be exhausted. Add or update the following parameter in your FE configuration (

fe.conf

):

properties
# Increase the number of worker threads for the StarMgr gRPC server (default is 1024)
# Supported in newer versions (v3.1.x+)
starmgr_grpc_server_max_worker_threads = 2048

3. Monitor FE Resource Usage Since scaling up triggers this issue, check the FE's performance metrics during the scale-up event: * CPU Usage: High CPU on the FE can delay gRPC request processing. * JVM GC: Check

fe.gc.log

to see if long "Stop-the-World" pauses are occurring, which would cause "Deadline Exceeded" for all incoming RPCs. * Network Latency: Ensure there is no network congestion between the new CN nodes and the FE. 4. Verify Port Connectivity Ensure the new CN nodes can communicate with the FE on the cloud-native metadata port (default is

6090

). If there is a firewall or security group delay, it could contribute to timeouts. Why this happens during scaling When you add a new CN node: 1. The node registers itself with the FE. 2. The FE may begin dispatching query fragments to the new node almost immediately. 3. The new node's cache is empty, so it must fetch metadata for every shard it processes from the FE. 4. This sudden burst of

GetShard

requests from the new node (combined with existing traffic) can overwhelm the FE's StarMgr processing queue if not tuned. If the issue persists after increasing the timeout, I recommend checking the FE logs (

fe.log

and

fe.warn.log

) for any

StarMgr

or

Leader

related errors around the same timestamp. References • deployment/environment_configurations.md • administration/stargo.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md