This message was deleted.

The warnings you are seeing in your Backend (BE) logs originate from bRPC, the underlying communication framework StarRocks uses for inter-node data transfer and coordination. What the log means *

EPOLLOUT

: This is a Linux system event indicating that a network socket is ready to send (write) data. *

Fail to wait EPOLLOUT

: The BE was trying to establish a connection or send data to another node (another BE or an FE), but the socket did not become ready within the allotted time. *

Connection timed out [110]

: This is the standard Linux error

ETIMEDOUT

. It means the connection attempt failed because the destination did not respond to the synchronization (SYN) packets in time. Common Causes These warnings usually point to network-level issues rather than a bug in StarRocks itself: 1. Network Connectivity/Firewalls: A firewall or security group may be blocking traffic on StarRocks ports (e.g.,

brpc_port

default 8060,

be_port

9050,

heartbeat_service_port

9050). 2. Target Node Overload: The destination node (another BE or FE) might be under extreme CPU or I/O load, preventing it from processing new network connections quickly enough. 3. Incorrect IP Configuration: If the

priority_networks

setting is not configured correctly in

be.conf

, the BE might be trying to connect to other nodes using an incorrect or unreachable network interface. 4. Network Congestion: High packet loss or latency in your physical or virtual network. Recommended Troubleshooting 1. Check Network Connectivity: Run

telnet <target_ip> <brpc_port>

(usually port 8060) from the BE showing the error to the destination nodes to see if the port is reachable. 2. Verify

priority_networks

: Ensure all nodes are using the correct internal IP addresses by checking the

priority_networks

parameter in

fe.conf

and

be.conf

. 3. Monitor Node Health: Check if any BE or FE nodes are experiencing high CPU usage, memory saturation, or long "Stop the World" (STW) Garbage Collection pauses. 4. Check Port Availability: Ensure that no other process is conflicting with StarRocks ports and that the OS

somaxconn

or

tcp_max_syn_backlog

limits aren't being hit (though this is less common). If these warnings occur only sporadically during heavy queries, they may be transient. However, if they are frequent and accompanied by query failures (e.g., "RPC transmit failed"), you likely have a network or resource bottleneck between your nodes. References • sql-reference/System_variable.md • faq/operation_maintenance_faq.md • loading/Flink-connector-starrocks.md • knowledge/usage/best-practice.md • faq/loading/Loading_faq.md • faq/loading/Broker_load_faq.md

@Rocky looks like C++ bug

While these warnings come from the C++ source code of bRPC (the networking library StarRocks uses), they are generally not considered a "bug" in the sense of a functional logic error. Instead, they are often a side effect of how bRPC handles network timeouts and idle connections, especially in containerized environments like Kubernetes. Why it looks like a "C++ bug" You are correct that this specific log message (

socket.cpp

) has been identified in the community as excessively noisy. There are a few reasons why this happens: 1. bRPC Idle Timeouts: If a connection is kept open but becomes idle, the environment (like a Kubernetes service, load balancer, or firewall) might silently drop the connection. When bRPC later tries to use or health-check that socket, it fails with a timeout, triggering this log. 2. Known Logging Issue: There is an open discussion in the StarRocks community (Issue #51912) regarding the "excessive" nature of this specific log. In many cases, the system is actually functioning correctly (retrying the connection), but the C++ library is being too "chatty" about standard network cleanup. 3. Environment Sensitivity: This warning became significantly more common starting with StarRocks version 3.1, often appearing when Pods are restarted or when there is transient network instability between Compute Nodes (CN) and Backends (BE). Recent Fixes/Improvements To address the "spammy" nature of these logs, StarRocks has been working on configuration improvements: *

brpc_idle_timeout_sec

: A new configuration (related to PR #67582) was proposed to better manage idle connections and prevent them from reaching a state where they trigger these

EPOLLOUT

timeout warnings. * Version 3.1+ Behavior: If you are on version 3.1 or later, you are more likely to see this. While it usually doesn't cause query failure on its own, it can clutter logs. Summary * Is it a bug? It's more of a logging/configuration issue than a crash-inducing bug. The C++ code is reporting a low-level network timeout that bRPC usually recovers from automatically. * Should you worry? If your queries are completing successfully, you can generally treat these as "noise." However, if you see them alongside query failures (e.g.,

RPC transmit failed

), it confirms a real network bottleneck or node-to-node connectivity issue in your infrastructure. References • faq/operation_maintenance_faq.md • sql-reference/System_variable.md • administration/management/resource_group.md • administration/Configuration.md • faq/loading/Loading_faq.md • GitHub Issue #51912: Too many logs in the BE/CN pods

@Rocky yes it appeared after restart

The fact that these appeared immediately after a restart strongly supports the theory that this is transient network "chatter" from the bRPC library rather than a functional bug. Why this happens after a restart When you restart a node (FE or BE), the following happens: 1. Stale Connections: Other nodes in the cluster still have open TCP connections to the "old" instance of the node you just restarted. When they try to send data or health checks to those stale connections, they time out, triggering the

Fail to wait EPOLLOUT

warning. 2. Service Readiness (Kubernetes): If you are running on Kubernetes, there is a delay between a pod starting and the K8s Service/Endpoints updating. During this window, other nodes might try to route traffic to a pod that isn't yet ready to accept bRPC traffic, leading to

Connection timed out

. 3. bRPC Handshake: When the restarted node comes back up, it aggressively tries to re-establish connections to every other node in the cluster. If several nodes are restarting at once (e.g., a rolling update), they may temporarily overwhelm each other's connection queues, causing these timeouts. Is it a problem? * If it stops after a few minutes: It is harmless. It’s just the bRPC library being very vocal while it synchronizes the cluster state and cleans up old sockets. * If it continues indefinitely: This suggests a node failed to join the cluster correctly. Check

SHOW BACKENDS

or

SHOW FRONTENDS

to ensure all nodes are

Alive: true

. Mitigation As mentioned in community discussions (like StarRocks Issue #51912), this log is known to be excessively noisy in version 3.1 and later. * Recent Versions: Newer versions of StarRocks have introduced

brpc_idle_timeout_sec

in the BE configuration to help the system close idle connections more gracefully, reducing these warnings. * Log Level: If the cluster is healthy, you can generally ignore these logs. They are a known artifact of how bRPC manages its socket pool during state changes. Summary: Your C++ "bug" suspicion is partially right—the library is indeed being "too loud" about standard network recovery—but it doesn't indicate a failure of the database logic itself. References • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • faq/Deploy_faq.md • GitHub Issue #51912: Too many logs in the BE/CN pods References • faq/Others.md • administration/Meta_recovery.md • faq/Deploy_faq.md • faq/operation_maintenance_faq.md