http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I'm using starrocks 4.0.8 shared ar...
# questions-and-troubleshooting
m
@Rocky I'm using starrocks 4.0.8 shared architecutre, I'm getting compute node already exists for cn-4, even though I did drop compute node and scale up pod still getting same. How can I fixt his? [Fri Oct 9 123437 IST 2026] Process conf file cn.conf ... [Fri Oct 9 123437 IST 2026] Add myself (starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050) into FE ... ERROR 1064 (HY000) at line 1: Compute node already exists with same host starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local and port 9050
r
Based on the error message and the behavior you're describing, the issue is likely that your CN startup script (or the entrypoint script in your container) is executing an 
ALTER SYSTEM ADD COMPUTE NODE
command blindly. Since the node record already exists in the FE's metadata, the FE returns an error, which causes your startup script to fail and prevents the CN process from starting. Here is why this is happening and how to fix it: 1. Why the "Already Exists" error persists In a Kubernetes environment (especially with StatefulSets), the FQDN 
starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local
remains constant for that pod slot. * Manual Drop: When you run 
DROP COMPUTE NODE
, the FE removes the entry. * Scale-Up/Restart: When the pod 
cn-4
starts, it runs a script that says "Add me to the FE". * Race Condition/Re-registration: If you have an Operator or a high-availability script running, it might have already re-added the node record the moment the pod became "known" to the system, or your 
DROP
command might not have used the exact string (host/port) that the script is using. * Metadata persistence: In StarRocks, once a CN is added, its metadata persists in the FE. If the pod crashes and restarts, it tries to add itself again. If the script doesn't check for existence, the 
ADD
command fails with 
1064
. 2. How to fix the startup failure You need to ensure the 
ADD COMPUTE NODE
command only runs if the node is not already registered, or allow it to fail gracefully without stopping the CN process. Option A: Modify the startup script (Recommended) If you are using a custom script, change the MySQL command to check if the node exists first. StarRocks 4.x supports 
ADD COMPUTE NODE
but does not always have an "IF NOT EXISTS" clause in all sub-versions. Instead, use a shell check:
Copy code
bash
# Example logic for your cn_entrypoint.sh
NODE_EXISTS=$(mysql -h ${FE_HOST} -P ${FE_QUERY_PORT} -u${USER} -e "SHOW COMPUTE NODES;" | grep "starrocks-cn-4.starrocks-cn-search")

if [ -z "$NODE_EXISTS" ]; then
    echo "Adding CN to FE..."
    mysql -h ${FE_HOST} -P ${FE_QUERY_PORT} -u${USER} -e "ALTER SYSTEM ADD COMPUTE NODE 'starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050';"
else
    echo "CN already exists in FE, skipping registration."
fi

# Now start the CN process
/opt/starrocks/cn/bin/start_cn.sh
Option B: Ignore the error in the script If you just want the pod to start regardless of whether the 
ADD
command fails, you can append 
|| true
to your registration command in the pod's 
command
or script:
Copy code
bash
mysql -h fe_host -e "ALTER SYSTEM ADD COMPUTE NODE ..." || true
3. Verify the Metadata To ensure there isn't a stale or "Dead" record causing issues, run this on your FE: 1. Check existing nodes:
Copy code
sql
    SHOW COMPUTE NODES;
2. Confirm the host string matches exactly: Ensure the 
IP
or 
Host
column matches 
starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local
exactly. If it appears as an IP, you must drop it by IP. If it appears as a FQDN, use the FQDN. 3. Clean Drop (if needed): If the node is listed but stuck in a "Dead" state:
Copy code
sql
    ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050";
Summary for StarRocks 4.0.8 (Shared Architecture) In shared architecture, CNs are stateless. You do not need to drop and re-add them every time a pod restarts. The best practice is to: 1. Add the CNs once. 2. Let the pods restart as needed. The CN will automatically reconnect to the FE using its heartbeat port (9050). 3. Only use 
ADD/DROP
when scaling the total number of nodes in the cluster (e.g., changing the replica count of the StatefulSet). If your automation is trying to 
ADD
on every pod start, update it to skip the command if 
SHOW COMPUTE NODES
already lists the host. References
m
@Rocky I've deployed starrocks in k8s, this is happening only for one pod cn-4, rest others on scale up/down those are fine but cn-4 is not coming up, always same error. already exists.
r
The error occurs because the StarRocks Frontend (FE) already has a record for 
cn-4
in its metadata, but the startup script inside the 
cn-4
pod is attempting to run 
ALTER SYSTEM ADD COMPUTE NODE
again. When this command fails with error 
1064
, the script exits, causing the pod to crash or fail to start. Since this is only happening for 
cn-4
, it's likely a metadata mismatch or a "ghost" record in the FE. Follow these steps to resolve it: 1. Identify the "Ghost" Record Run the following command on your FE (via MySQL client) to see exactly how 
cn-4
is registered:
Copy code
sql
SHOW PROC '/compute_nodes';
Look for the row where the IP or Host matches 
starrocks-cn-4...
. * Case A: It exists with a different ID. If you see multiple entries that look like 
cn-4
, you have a duplicate metadata issue. * Case B: It is marked as 
Alive: false
. The record is there but the node isn't heartbeating. 2. Cleanly Remove the Record Even if you already tried to "drop" it, the FE might be holding onto it if the identifier didn't match exactly (e.g., you dropped by IP but it's registered by FQDN). Run this command, using the exact string from the error message:
Copy code
sql
ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050";
Note: If you are using the StarRocks Operator, it might immediately try to re-add the node. If this happens, move to step 3. 3. Fix the Pod Startup Script The reason your pod won't come up is that the entrypoint script treats the "already exists" error as a fatal crash. To fix this for the long term (and prevent it happening to other pods), you should modify your deployment to ignore this specific error. If you are using a custom Helm chart or manual YAML, locate the 
command
or script that runs 
ADD COMPUTE NODE
and append 
|| true
to it:
Copy code
bash
# Example fix in a shell script
mysql -h fe_host -P 9030 -u root -e "ALTER SYSTEM ADD COMPUTE NODE '...'" || true
This allows the script to continue to the actual 
start_cn.sh
command even if the node is already registered. 4. Check for StarRocks Operator Conflict Are you using the StarRocks Operator? * If Yes: The Operator is responsible for adding/dropping nodes. You should not have a manual 
ADD COMPUTE NODE
command in your pod's startup script or 
ConfigMap
. If you do, they are fighting each other. Remove the manual SQL command from your container's entrypoint and let the Operator handle it. * If No (Manual Helm/YAML): This error is purely a script logic issue. The script should check 
SHOW COMPUTE NODES
before trying to 
ADD
, or simply ignore the error if it's already there. Why only 
cn-4
? In K8s StatefulSets, 
cn-4
always has the same hostname. If 
cn-4
was successfully added once, then crashed before it finished its setup, its name is now "taken" in the FE. While 
cn-0
through 
cn-3
might have succeeded and finished their scripts, 
cn-4
is now stuck because it's the only one retrying an 
ADD
for a name that is already in the database. Recommendation: 1. Manually 
DROP
the node one more time using the full FQDN string. 2. Update your CN startup logic to ensure it doesn't fail the whole pod just because the node is already registered. References • administration/management/enable_fqdn.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md
m
@Rocky how to update CN startup script when running in k8s with operator and how to change and restart for manually once
r
If you are using the StarRocks Operator, you should not typically need a manual script to add nodes, as the Operator is designed to manage this. However, if your deployment was set up with a custom script (often seen in older Helm charts or manual manifests), here is how you fix it. Part 1: The "Manual Once" Fix Before changing any scripts, you can unblock the pod by clearing the "stuck" metadata from the FE. This allows the pod's current script to succeed when it retries. 1. Connect to your StarRocks FE via a MySQL client (e.g., 
mysql -h <FE_IP> -P 9030 -u root
). 2. Find the exact record for cn-4:
Copy code
sql
    SHOW COMPUTE NODES;
3. Drop the specific node using the exact FQDN from your error message:
Copy code
sql
    ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050";
4. Restart the Pod:
Copy code
bash
    kubectl delete pod starrocks-cn-4 -n <your-namespace>
Since it's a StatefulSet, the pod will recreate. Because you dropped the node in Step 3, the 
ADD COMPUTE NODE
command in the script will now succeed. ────────── Part 2: Update the CN Startup Script (Permanent Fix) In an Operator-based deployment, the startup script is usually injected in one of two places: the StarRocksCluster YAML or a ConfigMap. Option A: Editing the StarRocksCluster CRD (Most Likely) Check your 
StarRocksCluster
resource to see if a custom 
command
or 
args
is defined. 1. Edit the resource:
Copy code
bash
    kubectl edit starrockscluster <cluster-name> -n <your-namespace>
2. Locate the 
starRocksComputeSpec
section. Look for 
command
or 
args
. It likely looks like this:
Copy code
yaml
    starRocksComputeSpec:
      command:
        • /bin/bash
        • -c
        • |
          # ... some code that runs mysql -e "ALTER SYSTEM ADD COMPUTE NODE ..."
3. Modify the line that runs the MySQL command. Append 
|| true
to the end of the MySQL execution line:
Copy code
bash
    mysql -h ${FE_HOST} -e "ALTER SYSTEM ADD COMPUTE NODE ..." || true
This tells the script: "Try to add the node; if the FE says 'Already Exists' (error), ignore it and keep going to start the CN process." Option B: Editing a ConfigMap If the 
command
in your YAML points to a script file (e.g., 
/opt/starrocks/scripts/cn_start.sh
), that script is likely stored in a Kubernetes ConfigMap. 1. Find the ConfigMap:
Copy code
bash
    kubectl get configmap -n <your-namespace>
    # Look for something like 'starrocks-cn-config' or 'cn-script'
2. Edit the ConfigMap:
Copy code
bash
    kubectl edit configmap <configmap-name> -n <your-namespace>
3. Update the logic to be "idempotent" (meaning it can run multiple times without failing):
Copy code
bash
    # Change this:
    mysql -h $FE_HOST -P $FE_PORT -e "ALTER SYSTEM ADD COMPUTE NODE ..."

    # To this:
    mysql -h $FE_HOST -P $FE_PORT -e "ALTER SYSTEM ADD COMPUTE NODE ..." || echo "Node already exists, proceeding..."
Why is this happening? In StarRocks Shared Architecture, Compute Nodes (CNs) are stateless. The StarRocks Operator or your script adds them to the FE's list of resources. * If a pod crashes after the 
ADD
command succeeds but before the CN process is fully healthy, the FE keeps the record. * When the pod restarts, it tries to 
ADD
itself again. * The FE rejects the second 
ADD
because the FQDN is already in the database. Summary: The best fix is to make your startup script ignore the error (
|| true
) so the CN process can actually start and begin communicating with the FE. References • administration/management/FQDN.md • deployment/deploy_starrocks_with_operator.md
m
@Rocky there is nothing in configmap or mysql add compute in starrockscnspec as well.
r
The error you're seeing—`Compute node already exists` followed by a pod crash—is a known issue in certain versions of the StarRocks Docker images and Operator. The reason you cannot find the 
mysql
command in your 
ConfigMap
or 
starRocksComputeSpec
is that it is hardcoded into the container's entrypoint script (
/opt/starrocks/cn_entrypoint.sh
). Why this is happening When the StarRocks Operator deploys a CN pod, it injects environment variables like 
FE_HOST
and 
FE_QUERY_PORT
. The script inside the image is programmed to: 1. See those variables. 2. Attempt to run 
ALTER SYSTEM ADD COMPUTE NODE ...
to register itself. 3. The Conflict: The StarRocks Operator also tries to register the node from the outside. If the Operator succeeds first, the pod's internal script fails with the "already exists" error. Because the script uses 
set -e
(fail on error), the entire pod crashes before the CN process can actually start. How to fix it 1. The Immediate "Manual" Fix To get 
cn-4
running right now without changing YAML, you need to catch the window where the FE doesn't know about it: 1. Drop the node from FE:
Copy code
sql
    ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050";
2. Immediately delete the pod so it restarts while the record is gone:
Copy code
bash
    kubectl delete pod starrocks-cn-4 -n <namespace>
Note: If the Operator is very fast, it might re-add the node before the pod starts, and you'll hit the error again. If this happens, use the permanent fix below. 2. The Permanent Fix (Bypass the script error) You need to tell the pod to ignore the registration error so it can proceed to start the CN process. You can do this by overriding the 
command
in your 
StarRocksCluster
YAML. Update your 
StarRocksCluster
resource (
kubectl edit src <name>
):
Copy code
yaml
spec:
  starRocksCnSpec:
    # ... existing config ...
    image: starrocks/cn-ubuntu:4.0.8 # Or your specific version
    command:
      - /bin/bash
      - -c
      - |
        # Run the official entrypoint but ignore any errors (like "already exists")
        # then explicitly start the CN process.
        /opt/starrocks/cn_entrypoint.sh || echo "Registration failed or already exists, starting CN anyway..."
        /opt/starrocks/cn/bin/start_cn.sh
3. Why only 
cn-4
? In a K8s StatefulSet, 
cn-0
through 
cn-3
likely succeeded because they were added for the first time during the initial deployment. 
cn-4
likely experienced a partial failure during its first-ever start—it successfully registered in the FE, but the pod itself died before it could mark itself as "started" in K8s. Now, every time it restarts, it tries to "Add" itself to a slot that it already occupies in the FE's metadata. Summary The script is inside the image and is being triggered by the 
FE_HOST
environment variable injected by the Operator. By overriding the 
command
to include 
|| true
or a manual start, you prevent the "already exists" error from being a fatal exit. Verify after restart: Once the pod stays "Running", check the status in StarRocks:
Copy code
sql
SHOW PROC '/compute_nodes';
The node should show 
Alive: true
. References • faq/shared_data_faq.md • administration/Meta_recovery.md • faq/Deploy_faq.md • deployment/deploy_manually.md
m
@Rocky I can see this in starrocks-cn statefulset spec: automountServiceAccountToken: false containers: - args: - $(FE_SERVICE_NAME) command: - /opt/starrocks/cn_entrypoint.sh env: - name: TZ value: Asia/Kolkata - name: MYSQL_PWD valueFrom: what to update in the commnand?
r
Since you are using the StarRocks Operator, you should modify the StarRocksCluster (SRC) custom resource rather than the StatefulSet directly (as the Operator will eventually overwrite manual changes to the StatefulSet). To fix the crash on 
cn-4
, you need to wrap the entrypoint script so that the "already exists" error (Exit Code 1064) does not kill the pod. Update your 
StarRocksCluster
YAML (specifically the 
starRocksCnSpec
section) with the following 
command
and 
args
override:
Copy code
yaml
spec:
  starRocksCnSpec:
    # ... your other config (image, requests, etc.)
    image: starrocks/cn-ubuntu:latest 
    command:
      - /bin/bash
      - -c
      - |
        # Run the entrypoint script but ignore errors so the pod doesn't crash
        # This allows the CN to proceed even if it's already registered in FE
        /opt/starrocks/cn_entrypoint.sh $(FE_SERVICE_NAME) || echo "Node already registered, proceeding to start..."
        
        # Ensure the CN process actually starts if the entrypoint exited early
        if ! pgrep -x "starrocks_be" > /dev/null; then
          /opt/starrocks/cn/bin/start_cn.sh --logconsole
        fi
Why this works: 1. 
|| echo ...
: The 
cn_entrypoint.sh
script is designed to fail (
set -e
) if the MySQL command to add the node returns an error. By adding 
|| echo
, you catch that failure, allowing the shell to move to the next line. 2. 
pgrep
check: If 
cn_entrypoint.sh
failed before it could launch the background process, this block manually triggers 
/opt/starrocks/cn/bin/start_cn.sh
. 3. 
--logconsole
: This ensures that logs are piped to 
stdout
so you can see them via 
kubectl logs
. To apply this manually "once" for cn-4: If you want to unblock just 
cn-4
immediately without a permanent YAML change: 1. Drop the node from the FE using a MySQL client:
Copy code
sql
    -- Use the exact FQDN from your error log
    ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050";
2. Delete the stuck pod:
Copy code
bash
    kubectl delete pod starrocks-cn-4 -n <your-namespace>
3. The Result: When the pod restarts, it will find that it is not in the FE metadata. The 
ADD COMPUTE NODE
command inside the script will succeed, and the pod will transition to 
Running
. Recommendation: Use the YAML update (Permanent Fix) to prevent this from happening to 
cn-5
, 
cn-6
, etc., during future scale-out operations or node restarts. References • deployment/deploy_starrocks_with_operator.md • administration/management/FQDN.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md References • administration/management/FE_configuration.md • deployment/deploy_manually.md • quick_start/shared-data.md
Open in Slack
PreviousNext
Powered by