@Rocky I am on starrocks 4.1.4, I have installed using operator helm with auto-scaling enabled on CN nodes. The problem is when my CN node downscales it doesn't de-register when I run `*SHOW* COMPUTE NODES;`I can still see the downscaled CN just

alive = false

and see following ErrMsg:

java.net.UnknownHostException: kube-starrocks-cn-1.kube-starrocks-cn-search.xeno-lakehouse.svc.cluster.local

. So basically CN pre-stop is not applying

ALTER SYSTEM DROP COMPUTE NODE "kube-starrocks-cn-1.kube-starrocks-cn-search.xeno-lakehouse.svc.cluster.local:9050";

query. This is leading to

Massive log spam on FE pods: failed to get right ip by fqdn kube-starrocks-cn-{6,7,8,9,10,11,12,13,14,15,16}

The behavior you are seeing—where downscaled CN (Compute Node) pods remain in the cluster metadata with

alive = false

and cause log spam—is currently a known behavior in the StarRocks Operator's lifecycle management. While the Operator handles the scaling of Kubernetes Pods, it does not automatically execute the

ALTER SYSTEM DROP COMPUTE NODE

SQL command during a scale-in event. This is consistent with how StarRocks handles BE nodes (to prevent accidental data loss), but it creates the log spam you mentioned for stateless CN nodes because the FE's Heartbeat Manager continues trying to resolve the FQDNs of the deleted Pods. Why this happens 1. Metadata Persistence: StarRocks FEs persist the list of all registered nodes in their image/metadata. Simply deleting a Pod does not remove it from this list. 2. DNS Resolution: When a Pod is deleted, its FQDN (e.g.,

kube-starrocks-cn-1...

) is removed from the Kubernetes DNS. The FE's frequent heartbeat checks then fail with

UnknownHostException

, leading to the "Massive log spam" you're seeing. 3. Pre-stop Limitation: The default

cn_prestop.sh

script in the operator image primarily focuses on graceful shutdown (finishing in-flight queries) rather than cluster de-registration. Recommended Solutions 1. Implement a Custom

preStop

Hook You can automate the de-registration by adding a custom

lifecycle

hook to your

starRocksCnSpec

in your Helm

values.yaml

. This hook will run the

DROP

command via the MySQL client before the Pod terminates.

yaml
starRocksCnSpec:
  lifecycle:
    preStop:
      exec:
        command:
        - /bin/sh
        - -c
        - |
          # Use the MySQL client to drop the node from the cluster
          # Replace <FE_HOST> and <FE_QUERY_PORT> with your FE service details
          mysql -h $FE_SERVICE_NAME -P 9030 -u root -e "ALTER SYSTEM DROP COMPUTE NODE \"${HOSTNAME}.${CN_SEARCH_SERVICE}.${POD_NAMESPACE}.svc.cluster.local:9050\";"
          # Then call the default graceful stop script
          /opt/starrocks/cn_prestop.sh

Note: Ensure the

mysql

client is available in your CN image (the default

starrocks/cn-ubuntu

usually includes it). 2. Use the StarRocks 4.0+ Blacklist Feature Since you are on version 4.1.4, you can leverage the Automatic Blacklist feature. While this doesn't remove the node from

SHOW COMPUTE NODES

, it prevents the FE from sending queries to these "dead" nodes, reducing query failures during the scale-down window. * Check if it is active:

SHOW COMPUTE NODE BLACKLIST;

* Note: This feature is primarily for query stability and may not fully stop the

UnknownHostException

log spam if the FE is still trying to send heartbeats. 3. Manual Cleanup (Post-Scale) If you do not want to automate the drop, you can periodically clean up disconnected nodes using a script or manual command:

sql
-- Identify disconnected nodes
SHOW COMPUTE NODES;
-- Drop them
ALTER SYSTEM DROP COMPUTE NODE "node_fqdn:9050";

Status of Official Fix There is an active community discussion and GitHub issue regarding the Operator's lack of automatic de-registration for CN nodes (e.g., Issue #62915). As of version 4.1, manual de-registration or a custom

preStop

hook remains the standard workaround for users requiring "clean" auto-scaling without FE log spam. References • deployment/sr_operator.md • administration/management/Scale_up_down.md • administration/stargo.md

@Rocky But this issue and it's related issues are already closed that it's resolved, then why am I still getting same errors

While some GitHub issues related to CN de-registration are marked as "closed" or "resolved," the resolution typically refers to the addition of the capability or fixes for specific edge cases (like upgrade restarts), rather than a default "hands-off" behavior that works in every environment. The reason you are still seeing

UnknownHostException

and log spam is that the StarRocks Operator does not, by default, perform database-level SQL commands (like

DROP COMPUTE NODE

) during a scale-in event. This is a safety measure to prevent accidental metadata corruption, even for stateless CNs. Why the error persists on 4.1.4 1. Metadata vs. DNS: When HPA scales down, Kubernetes deletes the Pod and its DNS record. However, the FE still has that FQDN registered in its internal metadata. 2. FE Heartbeat: The FE Heartbeat Manager periodically tries to resolve the FQDNs of all registered nodes. Since the Pod is gone, DNS resolution fails, leading to the

UnknownHostException

and the "failed to get right ip by fqdn" log spam. 3. Operator Scope: Most Operator "fixes" (like those in version 1.9.x) ensure that the CN process stops gracefully to finish queries, but they do not necessarily reach into the FE to run the

ALTER SYSTEM DROP

command unless you have configured the operator with the necessary database credentials and permissions. How to resolve this in your Helm deployment To stop the log spam and ensure nodes are removed from

SHOW COMPUTE NODES

, you must ensure the

DROP

command is executed during the Pod's termination. The most reliable way is to add a custom

preStop

hook in your `starRocksCnSpec`:

yaml
starRocksCnSpec:
  # ... other configs
  lifecycle:
    preStop:
      exec:
        command:
        - /bin/sh
        - -c
        - |
          # Use the internal service name of your FE
          FE_HOST="kube-starrocks-fe-service" 
          # Execute the drop command via mysql client
          # Note: Ensure the 'root' user has no password or use a secret for credentials
          mysql -h $FE_HOST -P 9030 -u root -e "ALTER SYSTEM DROP COMPUTE NODE \"${HOSTNAME}.${CN_SEARCH_SERVICE}.${POD_NAMESPACE}.svc.cluster.local:9050\";"
          # Allow time for the command to propagate before the process stops
          sleep 5
          /opt/starrocks/cn_prestop.sh

Alternative: FE Configuration If you want the FE to be more "forgiving" of missing DNS records (though this won't remove the nodes from the list), you can check the FE config

retry_interval_on_failure_second

, but this is generally not recommended as a fix for log spam. Recommendation If you believe this should be handled automatically by the operator version you are using, check your Operator logs. If the Operator is attempting to drop nodes but failing, it is usually due to: * Missing RBAC: The Operator needs permission to execute commands. * Missing Credentials: If your StarRocks

root

user has a password, the Operator cannot execute the

DROP

command unless that password is provided in the Operator's configuration (often via a Kubernetes Secret). For a clean auto-scaling experience without log spam, the Custom preStop Hook remains the industry-standard workaround for the StarRocks Operator. References • deployment/sr_operator.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/stargo.md