http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am running version 4.0.10 in K8s ...
# questions-and-troubleshooting
y
@Rocky I am running version 4.0.10 in K8s is shared mode. I am seeing these messages in the log file of one of my two FEs:
Copy code
2026-05-17 07:30:04.892Z INFO (AuditEventProcessor|69) [AuditLog.log():73] |Timestamp=1779003004880|Client=10.191.167.207:55872|User=root|AuthorizedUser='root'@'%'|ResourceGroup=|Catalog=default_catalog|Db=|St
2026-05-17 07:30:04.893Z INFO (starrocks-mysql-nio I/O-3|127) [MySQLReadListener.handleEvent():66] Client closed connection: 17087081 remote=10.191.167.207:55872
2026-05-17 07:30:04.893Z INFO (starrocks-mysql-nio I/O-3|127) [ConnectScheduler.unregisterConnection():241] Connection closed. remote=10.191.167.207:55872, connectionId=17087081, qualifiedUser=root, user.currC
2026-05-17 07:30:04.937Z INFO (starrocks-mysql-nio I/O-3|127) [AcceptListener.handleEvent():80] Connection established. remote=/10.181.68.150:29406, connectionId=17087082
2026-05-17 07:30:04.937Z INFO (starrocks-mysql-nio-pool-490|310904) [AcceptListener.lambda$handleEvent$1():88] Connection scheduled to worker thread 310904. remote=/10.181.68.150:29406, connectionId=17087082
2026-05-17 07:30:04.938Z INFO (starrocks-mysql-nio-pool-490|310904) [MysqlChannel.fetchOnePacket():195] Receive packet header failed, remote 10.181.68.150:29406 may close the channel.
2026-05-17 07:30:04.970Z INFO (starrocks-mysql-nio I/O-1|125) [AcceptListener.handleEvent():80] Connection established. remote=/10.191.167.207:55892, connectionId=17087083
2026-05-17 07:30:04.971Z INFO (starrocks-mysql-nio-pool-490|310904) [AcceptListener.lambda$handleEvent$1():88] Connection scheduled to worker thread 310904. remote=/10.191.167.207:55892, connectionId=17087083
2026-05-17 07:30:04.971Z INFO (starrocks-mysql-nio-pool-490|310904) [LeaderOpExecutor.forward():234] Forward statement 37526 to Leader TNetworkAddress(hostname:kube-starrocks-fe-1.kube-starrocks-fe-search.data
2026-05-17 07:30:04.991Z INFO (starrocks-mysql-nio-pool-490|310904) [LeaderOpExecutor.execute():142] forwarding to leader get result max journal id: 5870128
2026-05-17 07:30:04.991Z INFO (starrocks-mysql-nio-pool-490|310904) [JournalObservable.waitOn():51] waiting for the observer to replay journal to 5870128 with timeout: 0 ms
2026-05-17 07:30:04.991Z INFO (starrocks-mysql-nio-pool-490|310904) [LeaderOpExecutor.execute():149] waiting for star mgr journal replay to 974853
2026-05-17 07:30:04.991Z INFO (starrocks-mysql-nio-pool-490|310904) [JournalObservable.waitOn():65] waiting for the observer to replay journal to 974853 with timeout: 0 ms
2026-05-17 07:30:04.991Z INFO (AuditEventProcessor|69) [AuditLog.log():73] |Timestamp=1779003004971|Client=10.191.167.207:55892|User=root|AuthorizedUser='root'@'%'|ResourceGroup=|Catalog=default_catalog|Db=|St
2026-05-17 07:30:04.992Z INFO (starrocks-mysql-nio I/O-1|125) [MySQLReadListener.handleEvent():66] Client closed connection: 17087083 remote=10.191.167.207:55892
2026-05-17 07:30:04.992Z INFO (starrocks-mysql-nio I/O-1|125) [ConnectScheduler.unregisterConnection():241] Connection closed. remote=10.191.167.207:55892, connectionId=17087083, qualifiedUser=root, user.currC
2026-05-17 07:30:04.993Z WARN (starrocks-mysql-nio-pool-490|310904) [MySQLReadListener.handleRequest():162] Exception happened in one session(com.starrocks.qe.ConnectContext@a2b44eb).
java.nio.channels.ClosedChannelException: null
    at sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:215) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:527) ~[?:?]
    at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:153) ~[xnio-nio-3.8.16.Final.jar:3.8.16.Final]
    at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150) ~[xnio-api-3.8.16.Final.jar:3.8.16.Final]
    at org.xnio.channels.Channels.writeBlocking(Channels.java:147) ~[xnio-api-3.8.16.Final.jar:3.8.16.Final]
    at com.starrocks.mysql.MysqlChannel.realNetSend(MysqlChannel.java:247) ~[starrocks-fe.jar:?]
    at com.starrocks.mysql.MysqlChannel.send(MysqlChannel.java:240) ~[starrocks-fe.jar:?]
    at com.starrocks.mysql.MysqlChannel.flush(MysqlChannel.java:264) ~[starrocks-fe.jar:?]
    at com.starrocks.mysql.MysqlChannel.sendAndFlush(MysqlChannel.java:332) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.finalizeCommand(ConnectProcessor.java:857) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1143) ~[starrocks-fe.jar:?]
    at com.starrocks.mysql.nio.MySQLReadListener.handleRequest(MySQLReadListener.java:152) ~[starrocks-fe.jar:?]
    at com.starrocks.mysql.nio.MySQLReadListener.lambda$handleEvent$0(MySQLReadListener.java:92) ~[starrocks-fe.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
Why is this happening ?
r
The log messages you are seeing indicate two different behaviors occurring within your StarRocks cluster. Both are common in Kubernetes environments, especially in Shared-data mode. 1. "Receive packet header failed" (K8s Health Checks)
Copy code
text
Receive packet header failed, remote 10.181.68.150:29406 may close the channel.
This is almost certainly caused by Kubernetes Liveness or Readiness probes. * Why: Kubernetes probes often perform a simple TCP socket check on port 
9030
. Because the probe only opens the connection and does not send a valid MySQL handshake packet, the FE times out waiting for the first packet and logs this message. * Is it an issue? No. This is harmless noise and can be ignored. 2. 
ClosedChannelException
and Metadata Sync Wait The more significant logs involve a client connection that fails during a forwarded operation:
Copy code
text
[LeaderOpExecutor.forward():234] Forward statement ... to Leader
[JournalObservable.waitOn():51] waiting for the observer to replay journal ...
java.nio.channels.ClosedChannelException: null
The Workflow: 1. Forwarding: The FE you are looking at is a Follower/Observer. When it receives a request (usually a write or a metadata query), it must forward it to the Leader FE (
kube-starrocks-fe-1
). 2. Consistency Wait: After the Leader executes the operation, the Follower FE must wait (
JournalObservable.waitOn
) until it has replayed the metadata journals to the same point as the Leader. This ensures the Follower doesn't return stale data. 3. Client Timeout: While the FE was waiting for the metadata to sync, the client (10.191.167.207) reached its own timeout limit and closed the connection. 4. The Error: When the FE finally finished syncing and tried to send the result back, it found the socket already closed, resulting in the 
java.nio.channels.ClosedChannelException
. Why is this happening? If this is happening frequently, it suggests that metadata synchronization is taking too long, exceeding your client's timeout. Common causes include: * Metadata Replay Lag: In Shared-data mode, if your object storage (S3/GCS/Azure) or the network between FEs is slow, replaying the metadata journals can take several seconds. * High Metadata Load: A high volume of DDLs (Create/Drop Table) or frequent 
INSERT
jobs can create a backlog of journal entries that Followers must process. * Client Timeout Too Low: Your application or SQL client may have a short timeout (e.g., 5-10 seconds) that isn't long enough to account for the FE forwarding and sync overhead. Recommended Actions: 1. Identify the Leader: Check the status of your FEs using 
SHOW FRONTENDS;
. If you connect directly to the LEADER FE, you will bypass the "forward and wait" overhead. 2. Check FE Health: Verify the network latency between your FE pods. High latency will directly increase the 
waitOn
time. 3. Increase Client Timeout: If your application can tolerate it, increase the query/connection timeout in your MySQL client or connection pool settings. 4. Monitor Journal Lag: Check the FE logs for "replay journal" speed. If the "max journal id" difference is consistently high, your FEs may need more CPU or faster disk access for their metadata directories. References • knowledge/usage/best-practice.md • administration/Meta_recovery.md
2 Views
Open in Slack
PreviousNext
Powered by