@Rocky what is the correct way to decommission compute nodes? I normally do this via editing the StarRocks Kubernetes Operator deployment. However I am seeing constant DNS errors in the FE log for the nodes killed by the Operator and would like to have them cease.

Hi, I'm Rocky! The behavior you're seeing—where the FrontEnd (FE) continues to log DNS resolution errors after you scale down Compute Nodes (CNs) via the Kubernetes Operator—is a known occurrence. While the Operator successfully removes the Pods, the FE maintains a registry of these nodes in its metadata. Because it still "expects" these nodes to be part of the cluster, it continues to attempt heartbeats to their now-deleted Fully Qualified Domain Names (FQDNs), resulting in the DNS errors you see. Here is the correct workflow to decommission CN nodes cleanly and stop those logs: 1. Identify the "Dead" Nodes After you have reduced the

replicas

in your

StarRocksCluster

YAML, the FE will eventually mark those nodes as

Alive: false

. You can see them by running:

sql
SHOW COMPUTE NODES;

Look for nodes where

Alive

is

false

and the

IP

or

Hostname

matches the Pods that were removed. 2. Manually Drop the Nodes Compute Nodes are stateless (they only hold local cache, not persistent data), so they do not support the

DECOMMISSION

command used for Backends (BEs). Instead, you must use the

DROP

command to remove them from the FE's metadata:

sql
ALTER SYSTEM DROP COMPUTE NODE "host:port";

* Host: Use the FQDN or IP shown in the

SHOW COMPUTE NODES

output (usually the K8s pod DNS like

node-name.service-name.namespace.svc.cluster.local

). * Port: Use the

HeartbeatPort

(default is

9050

). 3. Why this happens (and why it's manual) * Metadata Consistency: The StarRocks Operator manages the Kubernetes resources (Pods, Services), but the StarRocks FE manages the cluster's internal state. Currently, the Operator does not automatically issue

DROP

commands to the FE during scale-in to prevent accidental data loss in BE scenarios. * The "Deregistration" Bug: There is a known issue in some recent versions (specifically 3.4.x+) where the CN's internal graceful shutdown script might fail to deregister itself from the FE leader before the Pod terminates. This leaves the "ghost" entry in the FE. Recommended Workflow for Future Scaling: To avoid the logs entirely, you can actually perform the steps in this order: 1. Identify which Pod names will be removed (K8s usually deletes the highest-indexed Pods first, e.g.,

cn-2

if scaling from 3 to 2). 2. Execute the

ALTER SYSTEM DROP COMPUTE NODE

command for that specific Pod. 3. Update the

StarRocksCluster

manifest to reduce

replicas

. By dropping the node from the FE first, you tell the cluster to stop "looking" for that Pod before it actually disappears from the network, preventing the DNS log spam. References • deployment/sr_operator.md • administration/stargo.md • administration/management/FE_configuration.md • administration/management/Scale_up_down.md

@Rocky can commands like

SHOW COMPUTE NODES

be used in a table function to allow column selection?

No, you cannot directly use

SHOW COMPUTE NODES

or similar administrative commands within a table function (e.g.,

SELECT * FROM TABLE(SHOW COMPUTE NODES)

) to allow column selection or filtering. In StarRocks,

SHOW

statements are for display purposes only and do not return a result set that the SQL engine can treat as a table. However, there is a better, more "SQL-native" way to achieve this. The Correct Way: Use Information Schema Starting in StarRocks 3.x, most administrative and cluster metadata is exposed through the

information_schema

database. You can query these as standard tables, allowing you to use

SELECT

,

WHERE

, and

JOIN

. To perform column selection or filtering on your Compute Nodes, use the following:

sql
-- Select specific columns and filter for dead nodes
SELECT 
    ComputeNodeId, 
    IP, 
    HeartbeatPort, 
    Alive, 
    ErrMsg 
FROM information_schema.compute_nodes 
WHERE Alive = 'false';

Why use this for decommissioning? Since you are decommissioning nodes via the Kubernetes Operator and seeing DNS errors, you likely need to identify the exact

IP:HeartbeatPort

to run the

DROP

command. This table allows you to script the removal:

sql
-- Get the exact strings for your DROP commands
SELECT CONCAT(IP, ':', HeartbeatPort) AS node_to_drop
FROM information_schema.compute_nodes
WHERE Alive = 'false';

Alternative: SHOW PROC If your version of StarRocks does not yet have the

compute_nodes

view in

information_schema

, you can use the

SHOW PROC

system. While

SHOW PROC

itself also cannot be used in a

SELECT

, it is the internal metadata path that the FE uses.

sql
SHOW PROC '/compute_nodes';

Summary of Table Functions In StarRocks, the

TABLE()

function is specifically reserved for: 1. External Data: Using

FILES()

to query S3/HDFS/Local files directly. 2. Specific Analysis: Such as

TABLE(explain_analyze())

in certain diagnostic scenarios. 3. Table Value Functions (TVFs): For specialized metadata (like

TABLE(get_tablet_log(...))

). For node management and metadata, always look at

information_schema.compute_nodes

or *`information_schema.backends`*` first. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • sql-reference/System_variable.md • sql-reference/information_schema/fe_metrics.md • sql-reference/information_schema/be_metrics.md • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md