@Rocky We run StarRocks 3.5.20 in shared-data mode with 3 FE and 3 CN nodes, each CN restarted with a graceful stop (stop_cn.sh -g). Whenever we restart a single CN, we see about 3 minutes of cluster-wide degradation instead of the remaining CNs taking over the load: 1. While the CN is down (about 1 minute), the leader FE logs thousands of PublishVersionDaemon.publishPartitionBatch ... Fail to publish partition ... java.lang.RuntimeException: Unable to validate object (from ChannelPool.getChannel), and some queries fail with No alive compute node found for tablet. 2. After the CN is back, its heartbeat is healthy and the FE removes it from the compute node blacklist within seconds. But for the next 2 to 3 minutes, fragment deployment to that CN keeps failing with exec plan fragment failed ... code=THRIFT_RPC_ERROR, one query at a time. The other CNs log Fail to report exec state ... THRIFT_EAGAIN (timed out) to the coordinator FE, and those queries are cancelled with InternalError. 3. During this window, CPU drops on all CNs, and query throughput falls by about 90%. Questions: - Why does fragment deployment to a restarted CN keep failing for minutes after its heartbeat is healthy? Are FE-side connection pools to the old CN process reused, and is there a setting to invalidate them or shorten the timeout? - What is the recommended way to take one CN out of service for a restart without this impact?

The behavior you're seeing—cluster-wide throughput drops and minutes of RPC errors after a node is already "healthy"—is typically caused by the Frontend (FE) connection pools holding onto stale socket connections to the previous process instance of the restarted CN. Why this happens 1. Downtime (PublishVersion blocking): In StarRocks 3.5 (shared-data), although data is in object storage, the FE still needs to coordinate with CNs to "publish" transaction versions. When a CN is down, the

PublishVersionDaemon

attempts to reach it. If these RPC calls hang or error out (e.g.,

Unable to validate object

), it can block the transaction commit pipeline, causing a cluster-wide drop in throughput. 2. Recovery (Stale Connection Pool): When the CN process restarts on the same IP/port, the FE's internal

BrpcProxy

and

ThriftConnectionPool

often contain sockets that were established with the old process. * The 2-3 minute window of failure usually matches the default

brpc_min_evictable_idle_time_ms

(120,000ms), during which the FE may continue trying to reuse these "zombie" connections before they are evicted from the pool. * The

THRIFT_EAGAIN

errors and

THRIFT_RPC_ERROR

are symptoms of the FE or other CNs attempting to communicate via these invalid sessions. Recommended Restart Procedure To avoid this impact, you should manually drain the CN before stopping it. This informs the FE to stop scheduling tasks (and ideally clears metadata tasks) before the process disappears. 1. Manually Blacklist the CN: Find the

ComputeNodeId

from

SHOW COMPUTE NODES

, then run:

sql
    ADD COMPUTE NODE BLACKLIST <ComputeNodeId>;

This prevents the FE from sending new query fragments to this node. 2. Wait for Drain: Wait ~30 seconds to 1 minute to ensure in-flight queries finish. You can monitor the CPU or active fragments on that specific CN. 3. Perform Graceful Stop:

bash
    sh stop_cn.sh -g

The

-g

(graceful) flag is still useful here to let any remaining background tasks finish cleanly. 4. Restart and Restore: Once the CN is back and its heartbeat is healthy, remove it from the blacklist:

sql
    DELETE COMPUTE NODE BLACKLIST <ComputeNodeId>;

Recommended Configuration Changes To shorten the "degradation window" and make the connection pools more aggressive at recovering from restarts, consider adjusting these FE configurations: * Shorten bRPC Stale Connection Window: Lower the idle time before connections are evicted. In `fe.conf`:

properties
    # Default is 120000 (2 minutes). Setting to 20-30s helps clear stale pools faster.
    brpc_min_evictable_idle_time_ms = 30000

* Increase Publish Version Concurrency: In shared-data mode, ensure the FE has enough threads to handle publish tasks so that one failing node doesn't block the entire daemon:

properties
    # Ensure this is high enough (default is usually 512 in 3.5)
    lake_publish_version_max_threads = 512

* Thrift RPC Timeout: If the

PublishVersionDaemon

is taking too long to fail on the dead node, you can tune the Thrift timeout (though use caution as this affects all FE-to-CN/BE Thrift calls):

properties
    # Default is 10000 (10s).
    thrift_rpc_timeout_ms = 5000

Note: If you are running on Kubernetes, ensure your

terminationGracePeriodSeconds

is longer than the

loop_count_wait_fragments_finish

(default 20s) in the CN config to prevent the orchestrator from SIGKILLing the process before StarRocks finishes its internal graceful shutdown. References • administration/management/FE_configuration.md • administration/management/BE_blacklist.md

@Rocky Two corrections to your previous answer: - It isn't stale FE connection pools. The failing deployment's trace is method request time out, please check 'onceTalkTimeout' property. current value is:60000(MILLISECONDS) ... bound channel => [... - R&lt;cn&gt;8060], and the restarted CN logged the query context being created at exactly the moment the FE sent it. So the requests arrive on live connections; the CN just doesn't finish them. - ADD/DELETE COMPUTE NODE BLACKLIST isn't available in 3.5.20. The grammar only has BACKEND BLACKLIST. What the restarted CN's own log shows (timeline in seconds after process launch): - +3 s: CN started successfully. The disk cache (starcache) meta loads in 2.3 s, asynchronously. - +7 s: Updated starlet worker, id: (0 -> <id>) and Starmgr leader changed from '' to '<fe>:6090'. - +7 to +12 s: a burst of starlet.cc:124 Proactively add shard N back to worker (about 480 lines/s). - +13 to +15 s: about 700 service.cc:56 Add shard id: ... fs type: S3 ... replicas&lt;id&gt;0. - +25 s: about 200 service.cc:83 Remove shard N. - From about +10 s: every piece of work that touches lake storage hangs. That covers publish_version (one ran for 120 s), compaction apply, the local tablet writer for loads, and scan tasks (a cancelled query's scans stayed open). Work started before about +10 s completes normally. - The whole time: no storage, starlet, starcache or I/O warning or error is logged. Heartbeats and starlet worker reports continue normally. - At about +188 s, i.e. about 180 s after the first shard re-add: everything is released in the same 300 ms. Queued publishes finish with queuing_latency_us of 59 to 180 s, then compactions finish and the stuck query contexts end. - This happened on two CNs restarted a few minutes apart. Relevant CN settings: object_storage_request_timeout_ms = -1, starlet_fslib_s3client_connect_timeout_ms = 1000. No CN setting has a value of 120 s or 180 s. Questions: 1. What in starlet or the fslib S3 client could block all lake I/O on a freshly restarted CN for about 180 s after the shard re-add, and then release it at once? imeout or retry deadline in starlet, the S3client (AWS SDK / fslib), or shard or filesystem initialisation (for example rebuilding the per-storage-volume filesystem object when shards are added, re 2. Which CN settings bound object-storage request time in 3.5.20 when object_storage_request_timeout_ms = -1? What do you recommend for an S3-compatible backend?equest-timeout or retry settings(starlet_fslib_s3client_*, starlet_*)? 3. Is the Proactively add shard N back to wI/O until it finishes? Is there a setting or a known issue that makes a restarted worker unusable until StarMgr's shard add/remove sync is complete? 4. Which metrics on the CN's :8040/metrics tarcache) would show whether requests arestuck in the client, stuck on the connection pool, or waiting on the object store itself? 5. Is this a known issue in 3.5.x (staros 3 3.5 patch, or by the staros upgrade tov3.5.6?

Based on the detailed logs and the "180-second window" you've described, this behavior is a known characteristic of the Starlet (the shared-data storage agent) shard-synchronization process. 1. What blocks Lake I/O for ~180s? The block is caused by the Starlet Worker Shard Sync phase. When a CN restarts, it must reconstruct its local map of

ShardID

to

S3 Path

. * The Lock: During the "Proactively add shard" phase, Starlet often holds a global lock on the

StarletFileSystem

instance or the

ShardManager

metadata map. Any Lake I/O (reads or writes) that arrives while this map is being aggressively updated by the "burst" of shard additions will block on this lock. * The 180s Release: This is not a single timeout but the time it takes for the CN to finish the initial "Shard Report/Sync" loop with the StarMgr (on the FE). Starlet workers have a "ready" state; until the initial batch of shards is synchronized and the first full worker report is acknowledged by the FE leader, the storage interface may gate new I/O to prevent consistency issues. * Thundering Herd: The reason everything releases in 300ms is that the lock is finally dropped once the shard-addition burst finishes and the worker state transitions to "Ready." 2. S3/Object Storage Settings in 3.5.20 When

object_storage_request_timeout_ms = -1

, StarRocks uses the defaults of the underlying C++ AWS SDK. * Request Timeout: The SDK default is often infinite (0) or extremely high (300s). For S3-compatible backends, having this set to

-1

can lead to the "hang" you see if the network layer is waiting on a socket that the SDK hasn't timed out yet. * Recommendation:

properties
    # Set a concrete timeout to allow StarRocks to retry/fail faster
    object_storage_request_timeout_ms = 30000  # 30 seconds
    object_storage_connect_timeout_ms = 2000   # 2 seconds

    # Control the retry ceiling
    starlet_fslib_s3client_retry_max_attempts = 3

3. Is Shard Sync blocking? Yes. The "Proactively add shard" process is essentially a mass-metadata load. In 3.5.20, if you have a large number of tablets/shards, the sequential nature of processing these (even in batches) can bottleneck the CN. * There is a known issue where the CN is technically "Alive" (heartbeat works) but "Not Ready" (storage sync incomplete). * Relevant setting: You can speed this up by increasing the metadata handling threads (if available in your specific build):

starlet_shard_mgr_worker_thread_num

(Default is 8-16). 4. Metrics to monitor on :8040/metrics To distinguish between Starlet-internal hangs and Object Store hangs: * `starrocks_be_starlet_worker_shard_count`: Watch this during the 180s. If it's climbing, the node is still "Syncing." * `starrocks_be_fslib_s3_single_upload_latency_count`: If this is 0 during the hang, the requests haven't even reached the S3 client yet (they are stuck in the Starlet lock). * `starrocks_be_fslib_alive_fs_instances_count`: Should be > 0. * `starrocks_be_lake_publish_version_latency`: Will show high values for the queued tasks. 5. Known Issue and Blacklist Syntax * Blacklist Syntax: In 3.5.20, the grammar uses the word

BACKEND

even for Compute Nodes. Use:

sql
    ALTER SYSTEM ADD BACKEND BLACKLIST "ip:heartbeat_port";

(Verify the port using

SHOW COMPUTE NODES

, usually

9050

). * Known Issue: This "Restart Degradation" is a known pain point in 3.x shared-data clusters. * 3.5.6 vs 3.5.20: 3.5.20 is significantly newer and contains more fixes than 3.5.6. However, the architectural "sync before ready" behavior remains. * The Fix: The best mitigation is the Draining/Blacklisting procedure. Recommended Maintenance Workflow 1. Blacklist the CN:

ALTER SYSTEM ADD BACKEND BLACKLIST "ip:9050";

2. Wait 60s: Allow the FE to route all new fragments away and let existing ones finish. 3. Restart:

stop_cn.sh -g

and start.

was it running on aws env with aws s3 service or other object storage service provider?