http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> How to resolve this error: Connecti...
# questions-and-troubleshooting
a
@Rocky How to resolve this error: Connection reset by peer [R1][E112]Not connected to
r
The error 
[R1][E112] Not connected to
combined with 
Connection reset by peer
indicates a failure in StarRocks' internal RPC communication (powered by brpc) between nodes—typically between a Frontend (FE) and a Backend (BE) or between two BEs. Common Causes 1. Node Downtime or Restart: The destination node (the IP/port mentioned in the error) might have crashed, restarted (e.g., due to OOM), or is currently unavailable. 2. Network Instability: Packet loss, high latency, or intermittent firewall interference can cause the TCP connection to be reset. 3. RPC Buffer Limits: High-concurrency or high-volume queries can saturate the RPC write buffer. If the unwritten data exceeds the limit, the connection may be dropped. 4. Resource Exhaustion: High CPU load or long JVM/BE Garbage Collection (GC) pauses can make a node unresponsive, leading to timeouts and connection resets. Recommended Troubleshooting Steps 1. Check Node Status Verify if the target BE/CN node is alive and stable. * Run 
SHOW BACKENDS;
or 
SHOW COMPUTE NODES;
in your MySQL client. * Check the 
be.out
or 
<http://be.INFO|be.INFO>
logs on the target node for signs of crashes (e.g., 
Segmentation fault
or 
OOM Killer
). 2. Verify Network Connectivity Ensure the nodes can communicate over the brpc_port (default 
8060
or 
8059
for some versions). * Test the connection from the source node to the target using 
telnet <IP> <brpc_port>
. * Check for network drops using:
Copy code
bash
    netstat -s | grep -i LISTEN
    netstat -s | grep TCPBacklogDrop
3. Adjust RPC Buffer Limits If the error occurs during large queries or high-load scenarios, try increasing the maximum unwritten bytes for brpc in your 
be.conf
(or 
cn.conf
): * Add or modify: 
brpc_socket_max_unwritten_bytes = 4294967296
(sets it to 4GB, default is often 1GB). * Note: This requires a restart of the BE/CN node to take effect. 4. Check priority_networks If you have multiple network interfaces, ensure 
priority_networks
is correctly configured in both 
fe.conf
and 
be.conf
to prevent nodes from attempting to connect via the wrong IP address. 5. Connection Pooling (Advanced) In some high-concurrency environments, setting the connection type to pooled can help. Add this to `be.conf`: * 
brpc_connection_type = pooled
If the issue persists, examine the FE/BE logs specifically around the timestamp of the error to see if it was preceded by a "Memory limit exceeded" or "Full GC" message. References • faq/operation_maintenance_faq.md • knowledge/usage/best-practice.md • administration/Meta_recovery.md • faq/Others.md • faq/Sql_faq.md • unloading/Flink_connector.md
a
@Rocky I am using front end nodes and computenodes using shared architecture
r
In a StarRocks Shared-data architecture, the error 
[R1][E112] Not connected to
indicates a network communication failure between a Frontend (FE) and a Compute Node (CN), or between two CNs. Since you are using Compute Nodes, follow these specific troubleshooting steps: 1. Verify CN Node Health Because CNs are often scaled elastically or run in containers (like Kubernetes), they may restart due to OOM (Out of Memory) or liveness probe failures. * Check Status: Run 
SHOW COMPUTE NODES;
in your SQL client. Ensure all nodes show 
Alive: true
. * Check Logs: Inspect 
cn.out
and 
<http://cn.INFO|cn.INFO>
on the node mentioned in the error IP. Look for 
Segmentation fault
or 
OOM Killer
messages which would explain why the connection was reset. 2. Increase RPC Buffer Limits If this error occurs during large queries or high-concurrency loads, the default RPC buffer might be too small, causing the connection to drop. * Action: Edit your 
cn.conf
file and add/increase the following parameter:
Copy code
properties
    brpc_socket_max_unwritten_bytes = 4294967296
(Note: This sets the limit to 4GB. You must restart the CN for this to take effect.) 3. Check 
priority_networks
 In shared-data environments (especially Kubernetes), nodes often have multiple network interfaces. If the FE or CNs are trying to communicate via the wrong IP, the connection will be reset. * Ensure 
priority_networks
is defined in both 
fe.conf
and 
cn.conf
to match your internal network (e.g., 
10.0.0.0/16
). 4. Firewall and Port Access Shared-data clusters rely heavily on the brpc_port (default 
8060
). * Ensure that the security group or firewall allows traffic on port 
8060
between all FEs and CNs. * From an FE node, try: 
telnet <CN_IP> 8060
. 5. Shared Storage Connectivity While 
E112
is a node-to-node error, if a CN becomes unresponsive while waiting for a slow Object Storage (S3/GCS) response, it can eventually trigger RPC timeouts. * Ensure your CNs have stable, high-bandwidth access to your storage volume. * Check if there are "Timeout" or "Deadline Exceeded" messages in the 
<http://cn.INFO|cn.INFO>
logs. If the error persists: Check the FE log (
fe.log
) for "connect processor exception." If the error happens during a specific large query, try setting 
SET pipeline_dop = 2;
(or a lower value) for that session to reduce the pressure on the RPC layer-to-layer data transmission. References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md
a
@Rocky I disabled this: UPDATE information_schema.be_configs SET VALUE = 'false' WHERE NAME = 'enable_connector_sink_spill'; but still getting the same error ERROR 5609 (22000): load channel: 019fbaa4-3216-76d5-9b53-3e872f375e16 was aborted at 1785542783, reason: Cancelled: Cancelled by pipeline engine, reason: Internal error: [E104]Fail to read from Socket
r
The error 
[E104] Fail to read from Socket
combined with 
load channel aborted
indicates that a connection was unexpectedly closed while StarRocks was trying to transfer data between nodes. This is common during heavy write operations (like 
INSERT INTO
or exporting to external tables) when a node becomes unresponsive or crashes. Since you are using a Shared-data architecture with Compute Nodes (CN) and mentioned 
enable_connector_sink_spill
(a feature introduced in v4.0.0 for writing to external tables like Iceberg), here are the steps to resolve this: 1. Check for CN Node Crashes The 
[E104]
error most often occurs when the "peer" node (the destination CN) disappears. * Check logs: On the CN node mentioned in the error IP, check 
cn.out
and 
<http://cn.INFO|cn.INFO>
. * Look for OOM or Segfaults: Look for "Out of memory" or "Segmentation fault." If the CN is crashing during a heavy load, it will drop all active sockets. * Verify status: Run 
SHOW COMPUTE NODES;
and check if any nodes have a recent 
LastStartTime
. 2. Increase RPC Write Buffer (Critical) When transferring large amounts of data (especially to external sinks), the default RPC buffer (1GB) can become saturated, leading to socket drops. * In your 
cn.conf
, add or update:
Copy code
properties
    brpc_socket_max_unwritten_bytes = 4294967296
* Note: You must restart the CN for this to take effect. This is the most common fix for 
E104
and "overcrowded" errors in StarRocks. 3. Reduce Parallelism (Pipeline DOP) If the cluster is overwhelmed by the number of parallel data streams, the pipeline engine may cancel tasks. * Try reducing the degree of parallelism for the session:
Copy code
sql
    SET pipeline_dop = 4; -- Or even 2 to test stability
4. Memory and Spilling You mentioned disabling 
enable_connector_sink_spill
. This parameter is intended to prevent OOM when writing to Iceberg by spilling to disk. * If you disabled it and are still getting errors, your CN nodes might be running out of memory (OOM) because they are now trying to keep all that data in RAM. * Recommendation: Re-enable it (
SET enable_connector_sink_spill = true
) but ensure the CN has enough local disk space for spilling (check 
spill_local_storage_dir
in 
cn.conf
). 5. Check Network & Firewall If the load takes a long time, an intermediate firewall or load balancer might be killing "idle" TCP connections. * Check if there are any network timeouts between the CNs and the Object Storage (S3/HDFS). * Ensure 
priority_networks
is set correctly in 
cn.conf
so nodes don't try to communicate over the wrong interface. Summary Checklist 1. Restart CN nodes after setting 
brpc_socket_max_unwritten_bytes = 4294967296
. 2. Lower 
pipeline_dop
 to reduce the number of concurrent sockets. 3. Monitor CN memory usage; if it spikes to 90%+ just before the error, it's an OOM issue. References • loading/Flink-connector-starrocks.md • administration/management/resource_management/spill_to_disk.md • knowledge/usage/best-practice.md • sql-reference/System_variable.md
a
@Rocky For Non nvme: sys_log_level = INFO # ports for admin, web, heartbeat service thrift_port = 9060 webserver_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 # Shared-data mode configuration starlet_port = 9070 storage_root_path = /opt/starrocks/cn/storage and for nvme sys_log_level = INFO # ports for admin, web, heartbeat service thrift_port = 9060 webserver_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 # Shared-data mode configuration starlet_port = 9070 # Use local NVMe SSD (RAID0) for cache — mounted by Karpenter instanceStorePolicy storage_root_path = /mnt/k8s-disks
r
Based on your configuration and the specific errors (
E104
and 
E112
), you are likely facing a scenario where your Compute Nodes (CN) are either crashing or becoming unresponsive during a heavy data operation. The fact that you have a mix of NVMe (ephemeral/Karpenter) and non-NVMe storage paths suggests a few specific areas to check: 1. Check for "Disk Full" on NVMe Cache In shared-data mode, StarRocks uses 
storage_root_path
to store the Data Cache (
starlet_cache
). If your NVMe disks (
/mnt/k8s-disks
) are smaller than the amount of data being processed, or if the cache is not being cleaned up quickly enough, the disk may fill up. * The Symptom: When a disk fills up, the CN can hang or crash, causing the FE to report 
Fail to read from Socket [E104]
. * The Fix: Explicitly limit the cache size in your 
cn.conf
for the NVMe nodes to ensure they don't exceed the available space on the mount:
Copy code
properties
    # Set this to ~80-90% of your actual NVMe disk size
    datacache_disk_size = 100G
2. The Risk of Disabling Spilling (
enable_connector_sink_spill
) You mentioned you disabled 
enable_connector_sink_spill
. This feature is designed to prevent Out of Memory (OOM) errors by spilling data to disk when writing to external sinks (like S3/Iceberg). * The Risk: By disabling it, your CN nodes must keep all data in RAM. If the load is large, the CN will likely hit its memory limit and be killed by the OS (OOM Killer). This results in an immediate connection reset (
E112
). * Recommendation: Re-enable spilling (
SET enable_connector_sink_spill = true
) and ensure your 
spill_local_storage_dir
in 
cn.conf
points to a path with enough space (on the NVMe nodes, this should be 
/mnt/k8s-disks/spill
). 3. Starlet/RPC Port Conflicts or Timeouts Since you are using a shared-data architecture, the communication between FE and CN happens over the 
brpc_port
(8060) and 
starlet_port
(9070). * Increase RPC Buffer: As noted previously, large data movements often exceed the default brpc buffer. Ensure all CN nodes have this in `cn.conf`:
Copy code
properties
    brpc_socket_max_unwritten_bytes = 4294967296
* Check Node Stability: Run 
SHOW COMPUTE NODES;
immediately after the error. If any node shows a very recent 
LastStartTime
, it definitely crashed. Check the 
cn.out
log on that specific node for 
Memory limit exceeded
or 
Segmentation fault
. 4. Mixed Node Types Performance Gap When you have a mix of standard disks and NVMe disks, StarRocks' pipeline engine might distribute work evenly, but the standard nodes may lag significantly behind the NVMe nodes. This "skew" can lead to timeouts in the query coordinator. * Troubleshooting: Check if the error consistently points to the IP of a "Non-NVMe" node or an "NVMe" node. If it's always the NVMe nodes, they are likely filling up their ephemeral storage or hitting OOM because they are processing data faster than the others. Summary Checklist 1. Check disk usage on 
/mnt/k8s-disks
during the load. 2. Verify CN logs (
<http://cn.INFO|cn.INFO>
and 
cn.out
) for any "Out of memory" messages. 3. Set 
datacache_disk_size
 to prevent the NVMe cache from filling the host disk. 4. Re-enable 
enable_connector_sink_spill
 if you are seeing high memory usage. References • administration/management/BE_configuration.md • deployment/shared_data/s3.md • administration/management/resource_management/spill_to_disk.md • using_starrocks/caching/block_cache.md References • deployment/shared_data/s3.md
a
@Rocky sys_log_level = INFO # ports for admin, web, heartbeat service thrift_port = 9060 webserver_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 # Shared-data mode configuration starlet_port = 9070 storage_root_path = /opt/starrocks/cn/storage is the one we are using for ebs volume for CN nodes what shall I add here?
r
For your Compute Nodes (CN) using EBS volumes (
/opt/starrocks/cn/storage
) in a shared-data architecture, you should add the following parameters to your 
cn.conf
to improve stability and prevent the 
[E104]
socket errors you are seeing. 1. Increase RPC Write Buffer (Essential) The 
E104: Fail to read from Socket
error during heavy loads or connector sinks is often caused by the default RPC buffer being too small for the volume of data moving between nodes.
Copy code
properties
# Increases the buffer for node-to-node communication to 4GB
brpc_socket_max_unwritten_bytes = 4294967296
2. Configure Local Spilling (Connector Sink) Since you are using 
enable_connector_sink_spill
(v4.0+ feature), you must ensure the CN node has a designated place on your EBS volume to spill data. If this isn't set, it may spill to the root partition, which is likely small and can cause the node to hang or crash when full.
Copy code
properties
# Directs spill data to your EBS volume instead of the root partition
spill_local_storage_dir = /opt/starrocks/cn/storage/spill
Note: Ensure the folder 
/opt/starrocks/cn/storage/spill
exists and is writable by the StarRocks user. 3. Limit the Data Cache On EBS volumes, if the Data Cache (
starlet_cache
) fills the entire disk, the node will become unresponsive. You should limit the cache to about 80-90% of your EBS volume's total capacity.
Copy code
properties
# Example: If your EBS volume is 200GB, set this to 160G
datacache_disk_size = 160G
4. Memory Management for "Shared-Data" Since CN nodes are stateless but perform heavy memory operations for external sinks: * Re-enable Spilling: You previously disabled 
enable_connector_sink_spill
. It is highly recommended to set this back to 
true
(
SET enable_connector_sink_spill = true;
). Disabling it forces the CN to keep all sink data in RAM, which is the most likely cause of the CN crashing (leading to your 
E104
error). * Check 
mem_limit
: Ensure 
mem_limit
in 
cn.conf
is set to a value that leaves some room for the OS (e.g., 90%). Summary of what to add to `cn.conf`:
Copy code
properties
# --- Connectivity & Stability ---
brpc_socket_max_unwritten_bytes = 4294967296

# --- Storage & Spilling ---
# storage_root_path = /opt/starrocks/cn/storage (already present)
spill_local_storage_dir = /opt/starrocks/cn/storage/spill
datacache_disk_size = <80%_of_ebs_size>G
After adding these, restart the CN nodes. Why this helps: EBS volumes have IOPS limits. If you don't limit the cache or define the spill path, the disk can hit its IO limit or capacity, causing the CN process to stall. When the process stalls, the network socket times out, resulting in the 
[E104]
error you reported. References • administration/management/BE_configuration.md • administration/management/resource_management/spill_to_disk.md • using_starrocks/caching/block_cache.md • deployment/shared_data/s3.md References • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md • sql-reference/System_variable.md • administration/management/FE_configuration.md
a
@Rocky May I know why is spil required do you know?
r
Spilling is a safety mechanism that prevents your StarRocks nodes (CN or BE) from crashing when they run out of memory. Here is why it is required: 1. Preventing Out-of-Memory (OOM) Crashes StarRocks is primarily an in-memory computing engine. When you perform heavy operations like large joins, sorts, or aggregations, StarRocks tries to keep all intermediate data in RAM to be fast. * Without Spilling: If the data size exceeds the node's memory limit, the operating system (or the StarRocks process itself) will kill the task to protect the system, leading to the 
Cancelled by pipeline engine
or 
Socket [E104]
errors you saw. * With Spilling: StarRocks writes the "excess" data to your EBS volume (
/opt/starrocks/cn/storage/spill
) and reads it back as needed. This allows the query to finish successfully, even if it's slower. 2. Handling "Connector Sinks" (Writing to Iceberg/S3) The specific parameter you mentioned, 
enable_connector_sink_spill
, is even more specialized: * Stability: When writing massive amounts of data to external tables (like Iceberg), the CN node must buffer data before sending it to S3. Without spilling, a large write can easily exhaust all CN memory. * File Optimization: Spilling prevents the "Small File Problem." If StarRocks is forced to flush data to S3 every time memory gets tight (without a spill disk), it would create thousands of tiny files, making your Iceberg table extremely slow to read. Spilling allows it to buffer larger chunks on your EBS disk before writing one clean, large file to the data lake. 3. Improving System Availability In a multi-tenant environment, one giant query shouldn't crash the whole cluster. Spilling ensures that "heavy" tasks can run in the background using disk space, leaving enough RAM for smaller, high-priority queries to remain fast. Summary You require spilling when your data volume is larger than your available RAM. Since you are experiencing socket errors and crashes, it's a strong sign that your current operations are exceeding the RAM on your CN nodes, making spilling (and a properly configured 
spill_local_storage_dir
on your EBS) essential for stability. References • administration/management/BE_configuration.md • sql-reference/System_variable.md • administration/management/FE_configuration.md • administration/management/resource_management/spill_to_disk.md
2 Views
Open in Slack
PreviousNext
Powered by