@Rocky ADLINE_EXCEEDED: CallOptions deadline exceeded after 29.999993370s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[remote_addr=127.0.0.1/127.0.0.1:6090]]] 2026-08-20 161216.757Z ERROR (thrift-server-pool-98|2942) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] Caused by: java.net.SocketException: Broken pipe at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?] at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?] at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?] at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0] ... 6 more 2026-08-20 161216.757Z WARN (thrift-server-pool-98|2942) [TIOStreamTransport.close():153] Error closing output stream. java.net.SocketException: Socket closed at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(NioSocketImpl.java:165) ~[?:?] at sun.nio.ch.NioSocketImpl.beginWrite(NioSocketImpl.java:371) ~[?:?] at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:416) ~[?:?] at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?] at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?] at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?] at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-08-20 161216.757Z ERROR (thrift-server-pool-99|2943) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] Caused by: java.net.SocketException: Broken pipe at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?] at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?] at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?] at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0] ... 6 more 2026-08-20 161216.758Z WARN (thrift-server-pool-99|2943) [TIOStreamTransport.close():153] Error closing output stream. java.net.SocketException: Socket closed at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(NioSocketImpl.java:165) ~[?:?] at sun.nio.ch.NioSocketImpl.beginWrite(NioSocketImpl.java:371) ~[?:?] at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:416) ~[?:?] at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?] at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?] at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?] at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-08-20 161218.325Z WARN (port-connectivity-checker-3|1652) [PortConnectivityChecker.isPortConnectable():113] socket connection to starrocks-fe-1.starrocks-fe-search.trino.svc.cluster.local:9020 failed, reason: Connection refused 2026-08-20 161218.325Z WARN (starrocks-http-pool-4|2719) [HttpServerHandler$RequestHandlingWatch.finish():254] receive slow http request. uri: /api/transaction/commit?, startTime: 1787242286029, latency: 52295 ms 2026-08-20 161218.325Z WARN (PortConnectivityChecker|18) [PortConnectivityChecker.runAfterCatalogReady():90] checking for connectivity of starrocks-fe-1.starrocks-fe-search.trino.svc.cluster.local:9020 failed, not open 2026-08-20 161218.325Z WARN (starrocks-http-pool-2|2706) [TransactionLoadAction.executeWithoutPassword():238] com.starrocks.common.StarRocksException: Can not abort COMMITTED transaction 8361595, label is rw-txn-5a8c7571-cf18-4ff6-a37b-f8dc6239d6e1-1787242280555492 at com.starrocks.http.rest.transaction.TransactionWithoutChannelHandler.handleRollbackTransaction(TransactionWithoutChannelHandler.java:141) at com.starrocks.http.rest.transaction.TransactionWithoutChannelHandler.handle(TransactionWithoutChannelHandler.java:80) at com.starrocks.http.rest.TransactionLoadAction.executeTransaction(TransactionLoadAction.java:254) at com.starrocks.http.rest.TransactionLoadAction.executeWithoutPassword(TransactionLoadAction.java:227) at com.starrocks.http.rest.RestBaseAction.execute(RestBaseAction.java:187) at com.starrocks.http.rest.RestBaseAction.handleRequest(RestBaseAction.java:114) at com.starrocks.http.HttpServerHandler.lambda$handleActionAsync$0(HttpServerHandler.java:140) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2026-08-20 161218.325Z WARN (starmgr-heartbeatmgr-0|135) [StarletAgent.heartbeat():105] caught GRPC exception when sending heartbeat to worker starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local:9070, io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: CallOptions deadline exceeded after 1.999990960s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[remote_addr=starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local/10.4.1.37:9070]]]. 2026-08-20 161218.325Z WARN (starrocks-http-pool-4|2719) [TransactionLoadAction.executeWithoutPassword():238] com.starrocks.common.StarRocksException: Can not abort COMMITTED transaction 8361590, label is rw-txn-8d66caae-093b-42cc-b713-ad98d6f02a17-1787242276271587 at com.starrocks.http.rest.transaction.TransactionWithoutChannelHandler.handleRollbackTransaction(TransactionWithoutChannelHandler.java:141) at com.starrocks.http.rest.transaction.TransactionWithoutChannelHandler.handle(TransactionWithoutChannelHandler.java:80) at com.starrocks.http.rest.TransactionLoadAction.executeTransaction(TransactionLoadAction.java:254) at com.starrocks.http.rest.TransactionLoadAction.executeWithoutPassword(TransactionLoadAction.java:227) at com.starrocks.http.rest.RestBaseAction.execute(RestBaseAction.java:187) at com.starrocks.http.rest.RestBaseAction.handleRequest(RestBaseAction.java:114) at com.starrocks.http.HttpServerHandler.lambda$handleActionAsync$0(HttpServerHandler.java:140) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2026-08-20 161218.326Z WARN (thrift-server-pool-49|2827) [FrontendServiceImpl.createPartitionProcess():2257] failed to add partitions com.starrocks.common.DdlException: Failed to create shard group. error: GRPCDEADLINE EXCEEDED CallOptions deadline exceeded after 29.999995330s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[remote_addr=127.0.0.1/127.0.0.1:6090]]] at com.starrocks.lake.StarOSAgent.createShardGroup(StarOSAgent.java:479) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createPartition(LocalMetastore.java:1907) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createPartition(LocalMetastore.java:1894) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createPartitionMap(LocalMetastore.java:1089) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.addPartitions(LocalMetastore.java:1353) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.addPartitions(LocalMetastore.java:971) ~[starrocks-fe.jar:?] at com.starrocks.service.FrontendServiceImpl.createPartitionProcess(FrontendServiceImpl.java:2251) ~[starrocks-fe.jar:?] at com.starrocks.service.FrontendServiceImpl.createPartition(FrontendServiceImpl.java:2086) ~[starrocks-fe.jar:?] at com.starrocks.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:6705) ~[starrocks-fe.jar:?] at com.starrocks.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:6682) ~[starrocks-fe.jar:?]

The errors you are seeing indicate a severe performance bottleneck or service failure within your StarRocks cluster's Shared-data (Cloud-native) architecture. Specifically, the FE StarMgr (StarOS) service, which manages shard metadata on port 6090, is failing to respond to requests within the 30-second deadline. This is causing a cascade of failures across DDL operations (partition creation), data loading, and heartbeats. Root Cause Analysis 1. StarOS/StarMgr Timeout (

127.0.0.1:6090

): * The

DEADLINE_EXCEEDED after 29.999s

error when calling

createShardGroup

means the FE's internal metadata service (StarMgr) is either overloaded or deadlocked. * Since it's calling

127.0.0.1

, it is trying to talk to the local StarMgr process running inside the FE. If this times out, the FE cannot complete table/partition creations. 2. FE Resource Saturation: * The

receive slow http request ... latency: 52295 ms

and

Broken pipe

(Thrift errors) strongly suggest the FE is experiencing extreme GC pauses (JVM) or CPU exhaustion. * When the FE hangs, it stops responding to RPCs on port

9020

(hence the

Connection refused

on other nodes) and HTTP requests for transaction commits. 3. Transaction Mismatch: *

Can not abort COMMITTED transaction

occurs because the client (loading tool) timed out waiting for the FE and sent an "abort" request. However, the FE eventually finished the "commit" in the background after the lag, leading to a state conflict. Recommended Troubleshooting Steps 1. Check FE JVM and Resource Health The most common cause is a "Stop the World" GC pause or CPU saturation on the FE. * Logs: Check

fe.gc.log

for long pauses (>1s). * Metrics: Check FE CPU and Memory usage. If memory is near the limit, increase

-Xmx

in

fe.conf

. * Thread Dump: If the FE is up but unresponsive, take a thread dump (

jstack <pid>

) to check for lock contention or deadlocks in

com.starrocks.lake.StarOSAgent

. 2. Tune StarMgr (Shared-Data specific) If you have a high volume of partition creations or DDLs, the StarMgr gRPC server might be hitting its thread limit. Add or increase these in `fe.conf`:

conf
# Increase gRPC worker threads for metadata management
starmgr_grpc_server_max_worker_threads = 2048

# If timeouts persist, you can try increasing the internal timeout (Default is often lower)
# starmgr_grpc_timeout_seconds = 60

3. Verify Connectivity to Port 6090 Ensure the FE is actually listening on port

6090

(controlled by

cloud_native_meta_port

in

fe.conf

). * Run

netstat -nlpt | grep 6090

on the FE node. * Check if any firewall or security group is blocking local gRPC communication. 4. Optimize Object Storage Access In Shared-data mode, shard creation involves metadata writes to your object storage (S3/GCS/OSS). * Check if your object storage provider is experiencing high latency. * If you are creating thousands of partitions at once, reduce the batch size. High bucket counts in partitioned tables can overwhelm the metadata service. 5. Check Cluster Connectivity The error

socket connection to starrocks-fe-1...:9020 failed

suggests that FE-1 might have crashed or its RPC service is down. Use

SHOW FRONTENDS;

to check the status of all FE nodes and ensure they are all

Alive

. Summary of Logs to Inspect *

fe.log

: Look for "StarOS" or "Starlet" related errors just before the timeouts. *

fe.warn.log

: Check for repeated "slow http request" or "lock" warnings. *

<http://be.INFO|be.INFO>

(on CN nodes): Check if Compute Nodes are also reporting errors connecting to the FE. References • administration/management/FE_configuration.md

All of my Fe pods are down they come up for a breif minute and go down again and I am also seeing these logs on restart @Rocky -08-20 161818.640Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242698640, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161823.641Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242703641, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161828.642Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242708642, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161833.644Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242713644, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161838.646Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242718646, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161843.647Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242723647, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161843.852Z WARN (PredicateColumnsDaemonThread|137) [StmtExecutor.execute():1078] execute Exception, sql: DELETE FROM statistics.predicate_columns WHERE fe_id='2' AND last_used < '2026-08-19 161843' org.apache.thrift.TException: java.net.ConnectException: Connection refused at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:60) ~[starrocks-fe.jar:?] at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:48) ~[starrocks-fe.jar:?] at com.starrocks.qe.LeaderOpExecutor.forward(LeaderOpExecutor.java:223) ~[starrocks-fe.jar:?] at com.starrocks.qe.LeaderOpExecutor.execute(LeaderOpExecutor.java:137) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.forwardToLeader(StmtExecutor.java:1321) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:848) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDMLOrControl(SimpleExecutor.java:94) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDML(SimpleExecutor.java:73) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsStorage.vacuum(PredicateColumnsStorage.java:284) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsStorage.vacuum(PredicateColumnsStorage.java:266) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsMgr.vacuum(PredicateColumnsMgr.java:189) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsMgr$DaemonThread.runAfterCatalogReady(PredicateColumnsMgr.java:240) ~[starrocks-fe.jar:?] at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?] 2026-08-20 161843.855Z WARN (PredicateColumnsDaemonThread|137) [StmtExecutor.execute():1078] execute Exception, sql: DELETE FROM statistics.predicate_columns WHERE fe_id='starrocks-fe-1.starrocks-fe-search.trino.svc.cluster.local_9010_1787235132619' AND last_used < '2026-08-19 161843' org.apache.thrift.TException: java.net.ConnectException: Connection refused at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:60) ~[starrocks-fe.jar:?] at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:48) ~[starrocks-fe.jar:?] at com.starrocks.qe.LeaderOpExecutor.forward(LeaderOpExecutor.java:223) ~[starrocks-fe.jar:?] at com.starrocks.qe.LeaderOpExecutor.execute(LeaderOpExecutor.java:137) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.forwardToLeader(StmtExecutor.java:1321) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:848) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDMLOrControl(SimpleExecutor.java:94) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDML(SimpleExecutor.java:73) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsStorage.vacuum(PredicateColumnsStorage.java:284) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsStorage.vacuum(PredicateColumnsStorage.java:270) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsMgr.vacuum(PredicateColumnsMgr.java:189) ~[starrocks-fe.jar:?] at com.starrocks.statistic.columns.PredicateColumnsMgr$DaemonThread.runAfterCatalogReady(PredicateColumnsMgr.java:240) ~[starrocks-fe.jar:?] at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?] 2026-08-20 161848.648Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242728648, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161852.386Z WARN (port-connectivity-checker-2|160) [PortConnectivityChecker.isPortConnectable():113] socket connection to starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local:9020 failed, reason: Connection refused 2026-08-20 161852.386Z WARN (port-connectivity-checker-0|158) [PortConnectivityChecker.isPortConnectable():113] socket connection to starrocks-fe-2.starrocks-fe-search.trino.svc.cluster.local:9020 failed, reason: Connection refused 2026-08-20 161852.386Z WARN (PortConnectivityChecker|18) [PortConnectivityChecker.runAfterCatalogReady():90] checking for connectivity of starrocks-fe-2.starrocks-fe-search.trino.svc.cluster.local:9020 failed, not open 2026-08-20 161852.386Z WARN (PortConnectivityChecker|18) [PortConnectivityChecker.runAfterCatalogReady():90] checking for connectivity of starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local:9020 failed, not open 2026-08-20 161853.649Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242733649, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161858.650Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242738650, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161903.651Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242743651, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161908.652Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242748652, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161913.653Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242753653, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER 2026-08-20 161918.654Z WARN (replayer|132) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787242758654, synchronized time: 1787242333625, has log: false, fe type: FOLLOWER

The logs indicate that your FE cluster is in a Deadlock or Quorum Loss state. No FE has been elected as the

LEADER

, and the

FOLLOWERS

are failing because they cannot find a leader to synchronize metadata with. 1. Identify the Crash Reason Before fixing the metadata, check why the pods are physically going down. Run:

bash
kubectl describe pod <fe-pod-name>
kubectl get pod <fe-pod-name> -o yaml

* OOMKilled: If the

lastState

is

Terminated

with reason

OOMKilled

, your FE needs more memory. Replaying large edit logs during startup is memory-intensive. Increase the FE RAM limit and

-Xmx

in

fe.conf

. * Liveness Probe Failure: If K8s is killing the pods because the probes fail, it’s because the FE is busy replaying logs and isn't "Ready" yet. Increase the

initialDelaySeconds

and

failureThreshold

of your liveness/readiness probes to give the FE 5-10 minutes to start. 2. Why "Meta out of date"? The

meta out of date

and

Connection refused

(port 9020) errors happen because: 1. The FE starts as a

FOLLOWER

. 2. It attempts to connect to the last known

LEADER

to sync logs. 3. The leader is either down or unreachable, so the follower stays in a "non-ready" state. 4. Since it's not ready, it fails its Kubernetes health check and is restarted, creating a Crash Loop. 3. Recovery Procedure (Quorum Recovery) If all FE pods are stuck and no Leader emerges, you likely have a BDBJE (metadata database) quorum issue. Step A: Stop all FE pods Scale the FE deployment/statefulset to 0. This ensures no locked metadata files exist.

bash
kubectl scale statefulset <fe-statefulset-name> --replicas=0

Step B: Identify the node with the latest metadata Check the PVs (Persistent Volumes) of your FEs. Look into the

meta/image

directory. The node with the largest

image.<number>

or the highest sequence in

meta/bdb

is your best candidate for the new leader (usually the one that was up longest). Step C: Force Start a Leader (Recovery Mode) Pick one FE pod (e.g.,

fe-0

) and force it to start as the Master to reset the quorum. 1. Edit the FE configuration (via ConfigMap or by manually editing the launch script in a temporary pod) to add:

conf
    metadata_failure_recovery = true

2. Scale the replicas to 1 (starting only

fe-0

). 3. Crucial: Watch the logs of

fe-0

. Once you see

Master/Leader started

, immediately remove the

metadata_failure_recovery = true

flag from the config and restart that pod normally. 4. Once

fe-0

is healthy and stable as Leader, scale the other replicas back up. They will then join as Followers and sync from

fe-0

. 4. Check Network Connectivity The logs show

starrocks-fe-0...:9020 failed

. Ensure that: * The Headless Service for FEs is working correctly. * Port 9010 (Edit log port) is open and reachable between all FE pods. If FEs cannot talk on 9010, they can never elect a leader. Summary Recommendation: 1. Increase FE Memory and Probe timeouts first (this fixes 80% of K8s FE loops). 2. If it still won't elect a leader, perform the Quorum Recovery (Step 3).

@Rocky These are the logs as of now since the pod has started, but its still not in running status - 2026-08-20 164240.102Z WARN (stateChangeExecutor|114) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 25620077 when applying visible log 2026-08-20 164240.102Z WARN (stateChangeExecutor|114) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 25371248 when applying visible log 2026-08-20 164240.102Z WARN (stateChangeExecutor|114) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 25373303 when applying visible log 2026-08-20 164240.102Z WARN (stateChangeExecutor|114) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 25368181 when applying visible log 2026-08-20 164240.102Z WARN (stateChangeExecutor|114) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 25369208 when applying visible log 2026-08-20 164240.102Z WARN (stateChangeExecutor|114) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 25376381 when applying visible log 2026-08-20 164240.102Z WARN (stateChangeExecutor|114) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 25374332 when applying visible log 2026-08-20 164307.842Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 164407.842Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 164507.854Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 164607.861Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 164707.867Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 164807.877Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 164907.879Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 165007.884Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details.

The logs indicate that your FE is healthy but extremely busy replaying metadata. The "ignored dropped partition" warnings are a normal part of the

LakeTableTxnLogApplier

process in StarRocks Shared-data (Cloud-native) mode; it's simply catching up on transaction logs for your lake tables. The reason your FE stays in

UNKNOWN

status and eventually goes down is likely a Kubernetes Probe Timeout. Kubernetes sees that the FE isn't "Ready" or "Live" yet and kills it, which resets the replay process and creates an infinite loop. Immediate Fix: Increase Kubernetes Probes You must give the FE enough time to finish replaying its logs. Replaying millions of edit logs can take 5–20 minutes depending on your disk speed and metadata volume. Modify your FE Deployment/StatefulSet YAML:

yaml
livenessProbe:
  initialDelaySeconds: 300  # Increase to 5-10 minutes
  periodSeconds: 30
  failureThreshold: 20      # Allow many failures during startup
readinessProbe:
  initialDelaySeconds: 300  # Increase to 5-10 minutes
  periodSeconds: 30
  failureThreshold: 20

Troubleshooting the "Stable State" Wait If the FE still doesn't become a LEADER/FOLLOWER after 20 minutes, check these specific causes mentioned in your logs: 1. Check for Quorum (Reason 2): If you have 3 FEs, at least 2 must be started and able to communicate to elect a Leader. If only

fe-0

is up, it will wait forever. Ensure all FE pods are attempting to start. 2. Clock Synchronization (Reason 4): StarRocks is very sensitive to time. Run

date

on all FE pods. If the difference is

5 seconds

, the BDBJE (metadata store) will refuse to elect a leader. Ensure your K8s nodes are running

ntpd

or

chronyd

. 3. Disk I/O Pressure: The

LakeTableTxnLogApplier

and BDBJE replay are heavy on disk I/O. If your metadata is on a slow network disk (like standard HDD or throttled cloud storage), replay will be very slow. Check the IOPS/latency on the FE metadata volume. 4. FE Metadata Cleanup: If your metadata has grown too large (too many edit logs without a checkpoint), the replay takes too long. Once you get one FE to successfully start as a Leader, it will eventually perform a Checkpoint, which creates a new

image

file and deletes old edit logs, making future restarts much faster. How to verify progress While the pod is starting, exec into the container and watch the metadata directory:

bash
# Watch the size of the bdb directory
du -sh /opt/starrocks/fe/meta/bdb/
# Watch the latest log file being updated
ls -lrt /opt/starrocks/fe/meta/bdb/ | tail -n 5

If the file sizes or names are changing, the FE is making progress. Do not restart it manually; let it finish. Summary Your FE is not crashing; it is being killed by Kubernetes while it is working. Increase your

initialDelaySeconds

and

failureThreshold

in your K8s spec to allow the metadata replay to complete. References • knowledge/trouble_shooting/fe_deadlock_problems.md • administration/management/FE_configuration.md • data_source/catalog/iceberg/iceberg_meta_table.md • administration/management/BE_configuration.md • data_source/catalog/iceberg/DDL.md

@Rocky it went into restart mode now - e":"starrocks-mysql-nio-pool-5","type":"INTENTION_SHARED","waitTime":41579,"queryId":"01a0201d-c4f9-7d3c-90aa-0bdb85436090"},{"id":2537,"name":"thrift-server-pool-70","type":"INTENTION_EXCLUSIVE","waitTime":41571},{"id":2475,"name":"starrocks-mysql-nio-pool-10","type":"INTENTION_SHARED","waitTime":41571,"queryId":"01a0201d-c501-71b9-aea1-f8c8ddafe4f4"},{"id":2458,"name":"starrocks-mysql-nio-pool-9","type":"INTENTION_SHARED","waitTime":41571,"queryId":"01a0201d-c501-780b-962b-2d2c730b4fc5"},{"id":33,"name":"tablet stat mgr","type":"INTENTION_SHARED","waitTime":39360},{"id":2453,"name":"starrocks-mysql-nio-pool-6","type":"INTENTION_SHARED","waitTime":37183,"queryId":"01a0201d-d624-7f46-b814-48291c65d80d"},{"id":1177,"name":"autovacuum-pool1-t5","type":"INTENTION_EXCLUSIVE","waitTime":35310},{"id":1183,"name":"autovacuum-pool1-t8","type":"INTENTION_SHARED","waitTime":35307},{"id":1181,"name":"autovacuum-pool1-t7","type":"INTENTION_SHARED","waitTime":35307},{"id":1179,"name":"autovacuum-pool1-t6","type":"INTENTION_SHARED","waitTime":35307},{"id":1170,"name":"autovacuum-pool1-t2","type":"INTENTION_SHARED","waitTime":35307},{"id":2538,"name":"thrift-server-pool-71","type":"INTENTION_EXCLUSIVE","waitTime":33362},{"id":2337,"name":"thrift-server-pool-23","type":"INTENTION_EXCLUSIVE","waitTime":33359},{"id":1172,"name":"autovacuum-pool1-t3","type":"INTENTION_SHARED","waitTime":27758},{"id":1168,"name":"autovacuum-pool1-t1","type":"INTENTION_SHARED","waitTime":27751},{"id":1173,"name":"autovacuum-pool1-t4","type":"INTENTION_SHARED","waitTime":27751},{"id":61,"name":"MemoryUsageTracker","type":"READ","waitTime":3194}]} 2026-08-20 170120.179Z WARN (starrocks-mysql-nio-pool-8|2455) [StmtExecutor.execute():935] Query 01a0201d-be46-7baa-9c47-e6f449b7cb01 failed. Planner profile : Planner: - -- Total[1] 1s675ms - -- Analyzer[1] 0 - -- Lock[1] 0 - -- AnalyzeDatabase[1] 0 - -- AnalyzeTemporaryTable[1] 0 - -- AnalyzeTable[1] 0 - -- Transformer[1] 0 - -- Optimizer[1] 1s671ms - -- MVPreprocess[1] 0 - -- MVTextRewrite[1] 0 - -- RuleBaseOptimize[1] 4ms - -- CostBaseOptimize[1] 1s665ms - -- PhysicalRewrite[1] 0 - -- DynamicRewrite[1] 0 - -- PlanValidate[1] 0 - -- TypeChecker[1] 0 - -- CTEUniqueChecker[1] 0 - -- InputDependenciesChecker[1] 0 - -- ColumnReuseChecker[1] 0 - -- ExecPlanBuild[1] 1ms - -- Pending[1] 0 - -- Prepare[1] 0 - -- Deploy[1] 9ms - -- DeployLockInternalTime[1] 9ms - -- DeploySerializeConcurrencyTime[2] 2ms - -- DeployStageByStageTime[6] 0 - -- DeployWaitTime[6] 6ms - -- DeployAsyncSendTime[4] 0 - -- DeployScanRanges[1] 0 - DeployDataSize: 43202 Reason:

The logs you've shared show a severe FE Metadata Lock Stall. Many threads (MySQL connections, Thrift RPCs, and Autovacuum tasks) are waiting for over 41 seconds to acquire a lock on the database metadata. Because the FE is locked, it cannot respond to Kubernetes health checks, leading to the restarts you are seeing. Root Cause: Metadata Lock Contention In StarRocks, metadata operations (like starting a transaction, creating a partition, or even planning a complex query) require a lock on the

GlobalStateMgr

. * The Waiters: Your log shows

thrift-server-pool

and

autovacuum-pool

threads waiting for

INTENTION_EXCLUSIVE

locks. * The Culprit: There is likely a single "holder" thread that has been holding a

SHARED

or

EXCLUSIVE

lock for a long time, blocking the entire queue. Common causes in Shared-data (Lake) clusters: 1. Massive Metadata Replay: If the FE was down and is now replaying a huge volume of "Transaction Logs" from the object store (as seen in your

LakeTableTxnLogApplier

logs), it holds locks while processing these updates. 2. Overloaded Autovacuum: The

autovacuum-pool

is responsible for cleaning up old data in the object store. If it's trying to clean thousands of partitions simultaneously, it can cause lock contention. 3. Heavy DDL/Schema Change: A long-running

ALTER TABLE

or a bulk

ADD PARTITION

operation. Recommended Actions 1. Disable/Throttle Autovacuum & Compaction Temporarily Since your logs show

autovacuum-pool

threads involved in the lock queue, try reducing the pressure on the FE metadata by adding these to your

fe.conf

and restarting:

conf
# Reduce parallel autovacuum tasks
lake_autovacuum_parallel_partitions = 2
# Increase the interval between autovacuum runs
lake_autovacuum_partition_naptime_seconds = 600
# Temporarily disable compaction if needed to stabilize
lake_compaction_max_tasks = 0

2. Increase Kubernetes Probe Thresholds (Required) The FE will eventually recover once it finishes replaying the logs or the blocking task completes, but Kubernetes is killing it too soon. * Set your

livenessProbe

and

readinessProbe

to have a

failureThreshold

of at least

30

and an

initialDelaySeconds

of

600

(10 minutes). This prevents K8s from restarting the FE while it's in the middle of a long metadata operation. 3. Identify the "Lock Holder" To find exactly which thread is holding the lock, you need a full thread dump while the FE is "stuck" but before it restarts. 1. Exec into the FE pod. 2. Run:

jstack -l <pid> > thread_dump.txt

3. Look for the thread that "locked" a

ReentrantReadWriteLock

. * Search for:

- locked <0x...>

* Search for:

GlobalStateMgr

or

Database.readLock/writeLock

in the stack trace. 4. Check FE JVM Memory Lock stalls can also be triggered by "Full GC" pauses. If the FE is replaying many logs, it consumes a lot of heap. * Check

fe.gc.log

. If you see "Full GC" occurring frequently right before the 41s lock wait, you must increase the memory (e.g., set

-Xmx

to 16G or 32G if your metadata is large). Summary of Configuration changes to stabilize: If you can edit the FE configuration (via ConfigMap), try these: 1.

lake_autovacuum_parallel_partitions = 4

(Down from 8) 2.

lake_compaction_max_tasks = 16

(Down from default adaptive) 3. Ensure

Xmx

is at least 8GB-16GB for stable metadata replay. 4. Increase K8s probe timeouts so the pod isn't killed during the 40s+ lock wait. References • administration/management/FE_configuration.md • administration/management/compaction.md • knowledge/usage/best-practice.md