@Rocky is it a known bug ?

mysql> select * from sys.fe_memory_usage;
ERROR 1064 (HY000): FE RPC failure, address=TNetworkAddress(hostname=<http://starrocks-fe-1.starrocks-fe-search.data-analytics.svc.kube.tab.com|starrocks-fe-1.starrocks-fe-search.data-analytics.svc.kube.tab.com>, port=9020), reason=Internal error processing listFeMemoryUsage: BE:10026, host: unknown
mysql> select * from sys.fe_locks;
ERROR 1064 (HY000): FE RPC failure, address=TNetworkAddress(hostname=<http://starrocks-fe-1.starrocks-fe-search.data-analytics.svc.kube.tab.com|starrocks-fe-1.starrocks-fe-search.data-analytics.svc.kube.tab.com>, port=9020), reason=Internal error processing listFeLocks: BE:10030, host: unknown

The error you are seeing is a known issue that typically occurs in multi-FE (Frontend) clusters, especially when deployed on Kubernetes. It indicates that the FE you are connected to is trying to collect data from other nodes (Frontends or Backends) to satisfy the

sys

table query but cannot resolve the network address of one of those nodes. Analysis of the Error *

FE RPC failure

: The FE is failing to communicate with another FE node via its internal Thrift RPC port (default

9020

). *

reason=Internal error processing listFeMemoryUsage: BE:10026, host: unknown

: *

listFeMemoryUsage

(and

listFeLocks

) are internal RPC methods used to aggregate system info across the cluster. *

BE:10026

: This is likely the internal ID of the node the FE is trying to reach. Note that even though it says "BE", in some error reporting paths, this prefix is used generically for node IDs (it might actually be the ID of a Frontend node). *

host: unknown

: This is the root cause. The FE's metadata contains a node with ID

10026

, but it doesn't have a valid IP or resolvable hostname for it. Common Causes 1. Node Metadata Inconsistency: A node (FE or BE) was added or dropped, but the metadata hasn't fully propagated or contains a "stale" entry with an empty/unresolvable host. 2. K8s DNS/Priority Networks: In Kubernetes, if

priority_networks

is not configured in

fe.conf

, FEs might register themselves with their internal Pod IP or a hostname that is not resolvable by other Pods. 3. Metadata Deadlock/Stall: If the Leader FE is under heavy load or having BDBJE (metadata sync) issues, the FOLLOWER FEs might have an inconsistent view of the cluster membership. Steps to Resolve 1. Check Node Status: Run the following commands and look for IDs 10026 and 10030:

sql
    SHOW FRONTENDS;
    SHOW BACKENDS;

Check if the

Host

column for these IDs contains a valid IP or a hostname that your FE pods can actually resolve (test this by exec-ing into an FE pod and pinging that address). 2. Verify Network Configuration: Ensure

priority_networks

is set in your

fe.conf

. In K8s, it should usually match the Pod CIDR (e.g.,

10.244.0.0/16

).

bash
    # Example in fe.conf
    priority_networks = 10.244.0.0/16

3. Restart Inconsistent FEs: If

SHOW FRONTENDS

shows an FE in a

false

Alive state or with an "unknown" host, try restarting that specific FE pod. If the node is stale/leftover from a previous deployment, you may need to manually drop it:

sql
    ALTER SYSTEM DROP FOLLOWER "host:port";

4. Version-Specific Bug: This error has been reported in various 3.x versions (e.g., 3.1.x, 3.2.x) when querying

information_schema

or

sys

tables. If your network is fine and nodes look healthy, check your StarRocks version. Improvements to RPC error handling for system tables were made in more recent patch releases (e.g., 3.3.x+). Suggested next step: Provide your StarRocks version and the output of

SHOW FRONTENDS;

if the problem persists. References • administration/management/FE_configuration.md • knowledge/trouble_shooting/fe_deadlock_problems.md • knowledge/trouble_shooting/fe_memory_problems.md