Hi I have shared nothing starrocks cluster with 3.4.5 running in k8s. sometimes I'm getting this error even though it is up. looks like kube-dns is taking time to do dns mapping.

error: Backend node not found. Check if any backend node is down.backend: [starrockscluster-be-2.starrockscluster-be-search.starrocks.svc.cluster.local alive: true inBlacklist: true] [starrockscluster-be-1.starrockscluster-be-search.starrocks.svc.cluster.local alive: true inBlacklist: true] 
2026-03-08 08:44:21.516Z INFO (starrocks-mysql-nio-pool-2965|86682) [DefaultCoordinator.cancel():985] cancel query 008cb31e-1acb-11f1-a725-b22fa7b41ce7 because Backend node not found. Check if any backend node is down.backend: [starrockscluster-be-2.starrockscluster-be-search.starrocks.svc.cluster.local alive: true inBlacklist: true] [starrockscluster-be-1.starrockscluster-be-search.starrocks.svc.cluster.local alive: true inBlacklist: true]

@Rocky what can I do to fix this issue?

The error you're seeing,

alive: true inBlacklist: true

, indicates that while the Frontend (FE) can reach the Backend (BE) via heartbeats, it has automatically added those BEs to its Internal Blacklist due to repeated RPC connection failures or query timeouts. In a Kubernetes environment, this is commonly triggered by DNS resolution latency or intermittent connectivity issues when the FE tries to resolve the BE FQDNs (e.g.,

starrockscluster-be-2.starrockscluster-be-search.starrocks.svc.cluster.local

). Here is how you can resolve and mitigate this issue: 1. Tune the FE Blacklist Thresholds Starting from v3.3.0, StarRocks automatically manages a BE blacklist. You can make the FE more tolerant of intermittent network/DNS flakiness by adjusting these dynamic FE configurations: *

black_host_connect_failures_within_time

: (Default: 5) Increase this to allow more failures before a node is blacklisted. *

black_host_history_sec

: (Default: 120) Decrease this to make the FE "forget" older failures faster. To apply these changes without restarting (SQL):

sql
ADMIN SET FRONTEND CONFIG ("black_host_connect_failures_within_time" = "50");
ADMIN SET FRONTEND CONFIG ("black_host_history_sec" = "30");

2. Verify and Clear the Blacklist You can check which nodes are currently blacklisted and why:

sql
-- View the current BE blacklist
SHOW BACKEND BLACKLIST;

-- If you need to manually remove a node from the blacklist (find BackendId from SHOW BACKENDS)
DELETE BACKEND BLACKLIST <BackendId>;

3. Optimize Kubernetes DNS (Recommended) Since you suspect

kube-dns

is the root cause, the most effective long-term fix in Kubernetes is to enable NodeLocal DNSCache. Standard

kube-dns

can suffer from packet loss and latency under high concurrency (the "conntrack race" issue). NodeLocal DNSCache runs a DNS caching agent on every node as a DaemonSet, significantly reducing resolution time and preventing the FE from failing to "find" the BEs. 4. Adjust DNS TTL (For v3.5.11+ or v4.0.4+) If you were on a later version, there is a

dns_cache_ttl_seconds

parameter. However, since you are on 3.4.5, the FE relies on the JVM's default DNS caching behavior. You can try adding the following to your FE JVM options (

fe.conf

) to ensure successful DNS lookups are cached longer:

bash
JAVA_OPTS="-Dnetworkaddress.cache.ttl=60"

Summary The quickest fix is to run the

ADMIN SET FRONTEND CONFIG

commands mentioned in step 1 to increase the failure threshold. This prevents the "Backend node not found" error from failing your queries when the nodes are actually healthy. References • administration/management/FE_configuration.md

@Rocky even after configuring

black_host_connect_failures_within_time and black_host_history_sec,

still getting same error.

It seems you are dealing with the BE/CN Blacklist feature introduced in StarRocks v3.3.0. This mechanism is designed to automatically remove problematic nodes from query execution when the FE detects connectivity issues. If you have already adjusted

black_host_connect_failures_within_time

and

black_host_history_sec

but the host is still blocked, here are the likely reasons and troubleshooting steps: 1. Check if the Node was Manually Blacklisted StarRocks will not automatically remove a node from the blacklist if it was added manually. * Action: Run the following command to check the `AddBlackListType`:

sql
    SHOW BACKEND BLACKLIST;
    -- or for Compute Nodes
    SHOW COMPUTE NODE BLACKLIST;

* Resolution: If the type is

MANUAL

, you must remove it manually:

sql
    ALTER SYSTEM DROP BACKEND "be_host:heartbeat_service_port" FROM BLACKLIST;

2. The Underlying Connection Issue Persists The automatic removal logic only works if the node passes connectivity checks. Within the window defined by

black_host_history_sec

, the number of connection failures must drop below the

black_host_connect_failures_within_time

threshold. * Action: Look at the

LostConnectionNumberInPeriod

in the

SHOW BACKEND BLACKLIST

output. If this number is high or increasing, the FE is still failing to reach the BE. * Check: Verify the network/firewall between the FE and the BE's

heartbeat_service_port

(default 9050) and

brpc_port

(default 8060). 3. FE-Specific Blacklists Each FE maintains its own blacklist and does not share it with other FEs. * Action: If you are using multiple FEs, you must run the

SHOW

commands on the specific FE that is throwing the error, or check all of them. 4. Is the error "Host is blocked because of many connection errors"? If the error occurs when you are connecting to the FE from your MySQL client (rather than during a query), this is a MySQL protocol security feature. * In standard MySQL, this is fixed by

FLUSH HOSTS

. * In StarRocks, this is typically governed by the

max_connect_errors

logic (though StarRocks usually doesn't block client IPs as aggressively as MySQL). If this is the case, try restarting the FE or check if you have a load balancer/proxy that is sending malformed packets. Recommended Troubleshooting Steps: 1. View the Blacklist status:

sql
    SHOW BACKEND BLACKLIST\G

Pay attention to

LostConnectionNumberInPeriod

. 2. Check FE Logs: Search for the string

blacklist

or the specific BE IP in

fe.log

to see the exact reason why the FE decided to keep the host in the blacklist. 3. Manual Override: If you are certain the node is healthy, try manually dropping it from the blacklist to see if it stays out:

sql
    ALTER SYSTEM DROP BACKEND "ip:port" FROM BLACKLIST;

Note: If you want to disable this feature entirely (not recommended for production stability), you can set the failure threshold to a very high number. References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • administration/management/BE_blacklist.md