@Rocky my CN node is getting this error repeatedly

09:56:38.576432 281470745629760 baidu_rpc_protocol.cpp:280] Fail to write into Socket{id=3521873190402 fd=3197 addr=10.110.23.14:51058:8060} (0xfffd7a929a00): Unknown error 1014 [1014]

The error code 1014 in StarRocks (via the bRPC library) corresponds to

EEOF

, which means the node "Got EOF" while trying to write to a socket. This indicates that the connection was closed by the remote peer (the node at the IP address

10.110.23.14

mentioned in your log) before the local CN node could finish sending its data. Common Causes 1. Remote Peer Crash or Restart: The node at

10.110.23.14

may have crashed (e.g., due to OOM) or was restarted, causing all its active connections to be dropped. 2. Network Instability: Firewalls, load balancers, or unstable network routes between the nodes may be terminating idle or long-running TCP connections. 3. Target Node Overload: If the peer node is under extreme pressure, its bRPC service may drop connections. While "overcrowding" usually triggers error

1011

, a hard reset or timeout can result in

1014

. 4. Resource Limits: If the remote node reached its file descriptor (FD) limit, it might immediately close new or existing connections. Recommended Troubleshooting * Check the Peer Logs: Log into the node at

10.110.23.14

and check its

<http://be.INFO|be.INFO>

(or

<http://cn.INFO|cn.INFO>

) and

be.out

logs at the exact timestamp (

09:56:38

) to see if it restarted or logged errors like

The server is overcrowded

or

Memory Limit Exceeded

. * Verify Network Stability: Check for packet loss or connection resets between these two nodes. Ensure there are no aggressive TCP timeout settings on intermediate network gear. * Check System Limits: Ensure the

ulimit -n

(file descriptors) is set to at least

65535

(preferably

1000000

) on all nodes. * Tune bRPC Settings: If this happens during heavy queries with large data shuffles, you might need to increase the socket buffer limit in `cn.conf`/`be.conf`:

properties
    # Increase the per-socket limit for unwritten outbound bytes (default is 1GB)
    brpc_socket_max_unwritten_bytes = 4294967296

Note: Changing this requires a restart of the CN/BE node. If the peer node is healthy and there are no network issues, this may be a transient error caused by a specific heavy query timing out. If it persists across all queries, it is likely a fundamental connectivity or resource issue. References • faq/Deploy_faq.md • knowledge/usage/best-practice.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • faq/loading/Broker_load_faq.md • faq/loading/Loading_faq.md