@Rocky

java.net.ConnectException: Connection refused

One of the BE node error messages. What to do

A

java.net.ConnectException: Connection refused

typically indicates that the Front End (FE) or a client is trying to connect to a Backend (BE) port, but the BE is not listening on that port. Here are the troubleshooting steps to identify and fix the issue: 1. Check BE Process Status Ensure the BE process is actually running on the target machine. * Command:

ps aux | grep starrocks_be

* Action: If it’s not running, start it using

./bin/start_be.sh --daemon

. 2. Inspect BE Logs If the process isn't running or keeps crashing, check the logs for the root cause. * File:

be/log/be.out

(Check this first for immediate crashes) * File:

be/log/be.INFO

or

be.WARNING

* Common Crash Cause (AVX2): StarRocks BE requires AVX2 instruction set support by default. If your CPU is older or you are running in a VM/Docker without AVX2 pass-through, the BE will crash immediately. * Check CPU support:

cat /proc/cpuinfo | grep avx2

* If missing, you must use a "no-avx2" version of StarRocks or enable it in your hypervisor. 3. Verify Network & Port Binding The BE might be running but bound to the wrong IP address or port. * Check Ports: Run

netstat -tunlp | grep <be_port>

(default ports: 9060, 9050, 8060, 8040) to see if they are listening. * Priority Networks: If the machine has multiple network interfaces, the BE might bind to the wrong IP. * Check

be/conf/be.conf

for the

priority_networks

setting. * Example:

priority_networks = 192.168.1.0/24

. * Firewall: Ensure the FE can reach the BE ports. Test from the FE node: *

telnet <BE_IP> 9060

or

nc -zv <BE_IP> 9060

4. Check Cluster Status Check how the BE is registered in the cluster. * SQL:

SHOW BACKENDS\G

* Look for: * `Alive`: If

false

, the FE cannot heartbeat the BE. * `ErrMsg`: Often contains more detail (e.g., "wrong cluster id"). * `IP`: Verify this matches the actual IP of the BE node. If it shows

127.0.0.1

and you are in a multi-node cluster, you must drop and re-add the BE using the correct IP. 5. Port Conflicts If the BE fails to start with an error like "service did not start correctly," another process might be using its ports. * Check for other services on ports 9060, 9050, 8060, or 8040. * If ports are occupied, change them in

be/conf/be.conf

. Summary Checklist: 1. Is the BE process running? 2. Does

be/log/be.out

show an AVX2 or "port already in use" error? 3. Can you

telnet

to the BE's

heartbeat_service_port

(default 9050) from the FE? 4. Does

SHOW BACKENDS

show the correct IP? References • faq/Deploy_faq.md • administration/management/BE_configuration.md • deployment/deploy_manually.md • administration/management/graceful_exit.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md

@Rocky It was an ephemeral issue. How do i prevent this from ever happening again

To prevent ephemeral "Connection refused" errors from impacting your cluster in the future, you should focus on three areas: automatic recovery, configuration tuning, and proactive monitoring. 1. Enable Automatic Process Recovery If the BE process stops briefly (due to an OOM, crash, or reboot), it should restart automatically without manual intervention. * Systemd: Configure StarRocks as a systemd service with

Restart=always

. This ensures the OS brings the BE back up immediately if it fails. * Kubernetes: If using the StarRocks Operator, K8s will automatically restart the pod and handle the liveness/readiness checks. * Supervisord: Use a process manager like

supervisord

to monitor and restart the

starrocks_be

process. 2. Tune FE-to-BE Resilience You can configure the Front End (FE) to be more tolerant of transient network glitches or brief BE restarts so they don't immediately fail your queries. * Increase Retries: In

fe.conf

, check/adjust

thrift_rpc_retry_times

(default is 3). Increasing this gives the FE more attempts to establish a connection before failing a task. * Blacklist Management: StarRocks has an automatic BE/CN blacklist feature. If a BE is flaky, the FE will temporarily route traffic away from it. * `black_host_connect_failures_within_time`: Number of failures before blacklisting (default 5). * `black_host_history_sec`: How long to keep history of failures (default 3600s). * Adjust Heartbeat Timeouts: If your network is high-latency, you can increase

thrift_connect_timeout_seconds

in

be.conf

to prevent premature "refused/timeout" errors during handshake. 3. Strengthen Network Stability * Priority Networks: Explicitly set

priority_networks

in both

fe.conf

and

be.conf

. This prevents StarRocks from binding to the wrong IP (like a Docker bridge or a management IP) after a machine restart. _ _Example:*

priority_networks = 10.10.10.0/24

* Port Reservation: Ensure the StarRocks ports (9060, 9050, 8060, 8040) are reserved or not used by other ephemeral processes on the machine to avoid port conflicts during restarts. 4. Monitoring and Alerting "Ephemeral" issues are often warnings of underlying resource exhaustion. * Prometheus & Grafana: Monitor the

starrocks_be_connections_total

and

starrocks_be_rpc_error_total

metrics. * Set Alerts: Create alerts for: * Process Down: Alert if a BE process is not running. * Connection Spikes: Alert if there is a sudden jump in

connection refused

or

thrift_rpc_error

. * Memory Pressure: Many ephemeral crashes are actually OOM kills. Monitor BE memory usage and ensure

mem_limit

in

be.conf

is set appropriately (usually 80-90% of total RAM). 5. Data Redundancy (High Availability) Ensure your tables use at least 3 replicas (

"replication_num" = "3"

). If one BE node has an ephemeral connection issue, the FE will automatically route queries to the other two replicas, making the issue invisible to your end users. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/BE_blacklist.md • faq/Deploy_faq.md • administration/monitor_metrics.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/BE_blacklist.md

@Rocky If I have one table with no replication and other with replication_num = 3 in the same shared nothing cluster, will 1 BE node cause every query to fail? Or the table with replication will still be queryable. Because 1/12 of my BE nodes went down and I couldn't query from the table with replication. I got the error message that one of the backend was not alive. They both share all 12 be node

In a StarRocks shared-nothing cluster, a table with

replication_num = 3

is designed to stay available even if one (or even two) BE nodes fail. However, if your query failed with a "backend not alive" error despite having 3 replicas, it is usually due to one of the following four reasons: 1. The Query Involved the 1-Replica Table If your query was a JOIN, UNION, or a subquery that involved both the 3-replica table and the 1-replica table, the entire query will fail if the 1-replica table has data on the dead BE. * Why: StarRocks must access all data required for the query plan. If even one tablet (from the 1-replica table) is unreachable, the execution plan cannot be completed, and the FE will return an error. 2. "Stale" or Unhealthy Replicas StarRocks only allows queries to read from healthy, up-to-date replicas. A replica is considered healthy only if its

version

matches the cluster's current

visible_version

. * Scenario: If your background data loading has been struggling recently, it's possible that only 1 out of the 3 replicas for certain tablets was successfully updated. * Result: If that one "up-to-date" replica happened to be on the BE that went down, the other 2 replicas are considered "stale" (outdated data). The FE will not route queries to stale replicas to avoid data inconsistency, leading to a "no alive backend" or "tablet has no healthy replica" error. * How to check: Run

SHOW TABLET <tablet_id>

for a tablet in that table to see the

Version

and

IsBad

status of all three replicas. 3. The "Detection Window" (Heartbeat Delay) The FE does not realize a BE is down instantly. It uses a heartbeat mechanism (default: every 5 seconds). * The Window: It usually takes 3 failed heartbeats (15 seconds) for the FE to officially mark a BE as

Dead

. * The Failure: If you run a query during those ~15 seconds, the FE might still try to send a query fragment to the dead BE. This results in a connection error. Once the 15 seconds pass and the FE sees the BE as

Alive: false

in

SHOW BACKENDS

, it will automatically begin routing queries to the other 2 healthy replicas. 4. Colocate Group Instability If your table is part of a Colocate Group (used to optimize joins by pinning related data to the same nodes), the failure of a BE can make the colocate group "unstable." * In some older versions or specific configurations, if a node in a colocate group is down, the FE may fail to generate a valid "colocate" plan for queries relying on that group until the group is marked as unstable and the FE falls back to a non-colocate plan. Summary Checklist 1. Was the 1-replica table part of the query? This is the most likely cause. 2. Wait 30 seconds: Does the query start working again once

SHOW BACKENDS

shows

Alive: false

? If yes, it was just the heartbeat detection delay. 3. Check Replica Health: Run

SHOW PROC '/statistic/<db_id>'

to see if there are "UnhealthyTablets". If you have unhealthy tablets, it means some replicas were already broken or out-of-sync before the node went down. Recommendation: For production environments, all tables should have

replication_num = 3

. Mixing 1-replica and 3-replica tables in the same cluster makes the cluster's availability only as strong as its weakest table. References • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • administration/Monitor_and_Alert.md • table_design/Colocate_join.md • faq/Deploy_faq.md • administration/management/BE_configuration.md References

@Rocky

Tablet lost replicas. Check if any backend is down or not. tablet_id: 8874196, replicas: 95876:8360/-1/8360/7309:NORMAL:ALIVE,. Check quorum number failed(OlapTableSink): BeReplicaSize:0, quorum:1

Transactions failing because of this error. What should I do. There was one BE node that temporarily went down. java.net.ConnectException: Connection refused. But it is alive again as show in

SHOW BACKENDS

The error

Check quorum number failed(OlapTableSink): BeReplicaSize:0, quorum:1

indicates that StarRocks cannot find any available, healthy Backend (BE) to write data to for a specific tablet. Because your

quorum

is 1, this confirms the table likely has

replication_num = 1

. Even though the BE is now "Alive" in

SHOW BACKENDS

, the loading process is still failing because the FE (Front End) hasn't yet verified that the tablet replica on that BE is healthy and up-to-date. Here is what is happening and how to fix it: 1. Wait for the Tablet Report When a BE restarts, it must report its thousands of tablets back to the FE. * Why it fails: Even if

SHOW BACKENDS

says the node is

Alive

, the FE might not have processed the "Tablet Report" for that specific tablet ID yet. Until it does, the FE assumes the data on that BE might be stale and refuses to write to it to prevent corruption. * Action: Usually, waiting 5–10 minutes allows the background reporting to finish. 2. Verify Replica Health Check the specific state of the tablet mentioned in your error: 1. Run:

SHOW TABLET 8874196;

2. In the output, copy the command under the

DetailCmd

column (it looks like

SHOW PROC '/dbs/...'

) and run it. 3. Check these columns: *

Version

: Does it match the other tablets in the table? *

IsBad

: Is this

true

? If so, the replica is corrupted. *

State

: It should be

NORMAL

. 3. Check for "Stale" Replicas If the BE was down while other data was being loaded into the cluster, the replica on that BE is now "stale" (its version is lower than the cluster's current version). * The Trap: Because you only have 1 replica, there is no "healthy" copy to sync from. StarRocks cannot automatically "fix" a single-replica tablet if it missed a version update while the node was down. * Action: If the version is stuck and won't advance, you may have to drop and recreate the affected partition (if data loss is acceptable) or restore from a backup. 4. Why this happened (and how to prevent it) You mentioned earlier that you have some tables with 3 replicas and some with 1. * The

quorum: 1

failure happens because for a 1-replica table, there is zero fault tolerance. If that one BE is down (or just restarted), all writes to that tablet will fail. * Prevention: Change your table properties to

replication_num = 3

.

sql
    ALTER TABLE <table_name> SET ("replication_num" = "3");

Note: You must have at least 3 BE nodes in your cluster for this to work. 5. Immediate Troubleshooting Commands If the issue persists and the BE is definitely healthy, you can try to trigger a manual repair/balance check (though FE usually does this automatically): * Check for cluster-wide issues:

SHOW PROC '/statistic';

* Look for "UnhealthyTablets". If the number is non-zero, navigate into that DB ID (e.g.,

SHOW PROC '/statistic/10123'

) to see which tablets are failing. Summary: Your transaction is failing because StarRocks is protecting data integrity by not writing to a replica it hasn't fully "checked in" yet after the restart. If it doesn't clear up in 10 minutes, the replica is likely version-mismatched due to the

replication_num = 1

setting. References • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR_TABLE.md • administration/Monitor_and_Alert.md References • administration/management/resource_management/Replica.md