@Rocky kushagra@Kushagras-MacBook-Pro ~ % kubectl logs starrocks-cn-1 -n trino -f [Thu Aug 20 130444 UTC 2026] Process conf file cn.conf ... [Thu Aug 20 130444 UTC 2026] Add myself (starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 1064 (HY000) at line 1: Compute node already exists with same host starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local and port 9050 [Thu Aug 20 130444 UTC 2026] run start_cn.sh

This error occurs because the startup script in the StarRocks Compute Node (CN) container automatically tries to register the node with the FrontEnd (FE) using an

ALTER SYSTEM ADD COMPUTE NODE

command. In a Kubernetes environment, when a pod restarts, it often retains the same FQDN (Fully Qualified Domain Name). Since the node was already registered in the FE's metadata during its first boot, the FE returns ERROR 1064 because it cannot add a node that already exists. Is this a problem? Usually, no. If you look at the last line of your log:

[Thu Aug 20 13:04:44 UTC 2026] run start_cn.sh

This indicates that the registration script encountered the error, ignored it, and proceeded to start the actual CN process. How to verify if the CN is working: 1. Check the Pod Status: Run

kubectl get pods -n trino

. If the pod status is

Running

, the CN has likely started successfully. 2. Check the StarRocks Cluster Status: Log into your StarRocks FE using a MySQL client and run:

sql
    SHOW COMPUTE NODES;

Look for the node

starrocks-cn-1...

and check if the

Alive

column is

true

. 3. Check CN Internal Logs: If the pod is crashing after the logs you shared, the actual error will be inside the CN's internal log files. You can check them with:

bash
    kubectl exec -it starrocks-cn-1 -n trino -- cat /opt/starrocks/cn/log/cn.INFO

(Note: Path may vary depending on your Docker image, typically

/opt/starrocks/cn/log/

or

/data/starrocks/cn/log/

). Summary The

ERROR 1064

is a harmless warning from the auto-registration script during a pod restart. You only need to investigate further if the pod enters a

CrashLoopBackOff

state or if

SHOW COMPUTE NODES

shows the node as

Alive: false

. References • administration/management/BE_configuration.md • quick_start/shared-data.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • deployment/deploy_manually.md • administration/Meta_recovery.md

CN1 and CN0 are alive CN2 is not @Rocky

If CN2 is showing as not alive while CN0 and CN1 are healthy, the issue is likely specific to that pod's network connectivity or the internal process state. Follow these steps to diagnose why CN2 is down: 1. Check the Error Message in StarRocks Run the following SQL command in your MySQL client:

sql
SHOW COMPUTE NODES\G

Find the row for CN2 and look at the

ErrMsg

column. This often tells you exactly why the FE thinks the node is down (e.g., "heartbeat timeout", "connection refused", or "mismatched cluster ID"). 2. Check the Pod and Container Status Check if the Kubernetes pod is actually running or stuck:

bash
kubectl get pods -n trino | grep starrocks-cn-2

* If it's in

CrashLoopBackOff

: Run

kubectl describe pod starrocks-cn-2 -n trino

and check the Events at the bottom. Look for OOMKilled (Out of Memory) or failing readiness probes. * If it's

Running

: The process might be alive, but the FE cannot talk to it. 3. Inspect CN2 Internal Logs Since the

kubectl logs

might only show the startup script, you need to look at the actual StarRocks engine logs inside the pod:

bash
# Check for any FATAL errors that caused a crash
kubectl exec starrocks-cn-2 -n trino -- cat /opt/starrocks/cn/log/cn.FATAL

# Check the last 100 lines of the INFO log for heartbeat or connection errors
kubectl exec starrocks-cn-2 -n trino -- tail -n 100 /opt/starrocks/cn/log/cn.INFO

Look for lines containing

heartbeat

or

failed to connect to FE

. 4. Verify Network Connectivity The FE must be able to reach CN2 on the

heartbeat_service_port

(default

9050

). From an FE pod, try to ping or reach CN2:

bash
kubectl exec -it <starrocks-fe-pod-name> -n trino -- curl -v starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:8040/api/health

_(Note: Replace 8040 with your

be_http_port

if different)._ Common Causes for Single-Node Failure: * Resource Limits: CN2 might have been assigned to a Kubernetes node with less available memory, triggering an OOM kill. * FQDN Issues: If using FQDNs, ensure CN2's hostname is correctly resolvable from the FE pods. * Cluster ID Mismatch: If CN2 was previously part of a different cluster, it might fail to join. This would be visible in

<http://cn.INFO|cn.INFO>

or

cn.WARNING

. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • administration/management/logs.md • deployment/sr_operator.md • administration/http_interface.md • administration/management/BE_configuration.md References • administration/management/logs.md • administration/http_interface.md • deployment/sr_operator.md • deployment/upgrade.md

@Rocky ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125936 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125938 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125940 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125942 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125944 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125947 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125949 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125951 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125953 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125955 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125957 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 125959 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 130001 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 130003 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 130005 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) ERROR 2003 (HY000): Can't connect to MySQL server on 'starrocks-fe-service:9030' (111) [Thu Aug 20 130007 UTC 2026] Add myself (starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local:9050) into FE ... ERROR 1064 (HY000) at line 1: Compute node already exists with same host starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local and port 9050 [Thu Aug 20 130007 UTC 2026] run start_cn.sh

The logs show that CN2 initially failed to connect to the FE (

starrocks-fe-service:9030

) for several minutes with a Connection Refused (111) error. It eventually succeeded at

13:00:07

, skipped registration (because the node already exists), and attempted to start the CN process (

run start_cn.sh

). If CN2 is still showing as "not alive" in

SHOW COMPUTE NODES

, it likely means the process crashed shortly after starting or there is a network issue preventing heartbeats. Please perform these three checks to find the root cause: 1. Check the

ErrMsg

in StarRocks Run this command in your MySQL client to see why the FE thinks the node is down:

sql
SHOW COMPUTE NODES\G

Look for CN2 and check the

ErrMsg

field. * If it says "heartbeat timeout", the FE cannot reach the CN. * If it says "mismatched cluster id", the CN was initialized with a different FE's metadata. 2. Check the CN Internal Logs (The most important step) The

kubectl logs

you shared only show the shell script output. The actual error causing a crash will be inside the CN's internal log files. Check these:

bash
# Check for fatal crashes
kubectl exec starrocks-cn-2 -n trino -- cat /opt/starrocks/cn/log/cn.FATAL

# Check for warnings or startup failures
kubectl exec starrocks-cn-2 -n trino -- tail -n 100 /opt/starrocks/cn/log/cn.WARNING

(Common errors:

Memory limit exceeded

,

Failed to initialize JNI

, or

Unable to bind to port

). 3. Check for OOM (Out of Memory) Since CN0 and CN1 are working, CN2 might have been scheduled on a Kubernetes node with less memory and got killed by the OS.

bash
kubectl describe pod starrocks-cn-2 -n trino

Look for "OOMKilled" in the "Last State" or "Events" section. Summary of what happened in your logs: * The Connection Refused (111) loop: This is normal during startup if the FE service or the network isn't fully ready yet. The script retries until it connects. * The 1064 Error: This is a harmless warning. It just means the CN was already registered in the FE's metadata from a previous attempt. * `run start_cn.sh`: This confirms the startup script finished. If the node is still dead, the failure happened inside the

starrocks_be

(CN) process itself. Next step: Run

kubectl exec starrocks-cn-2 -n trino -- tail -f /opt/starrocks/cn/log/cn.INFO

and look for any line that says

heartbeat

or

close

. References • deployment/deploy_manually.md • deployment/upgrade.md • deployment/downgrade.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md

@Rocky, error message - java.net.ConnectException: Connection refused

The

java.net.ConnectException: Connection refused

error is a heartbeat failure message generated by the FrontEnd (FE). Since the FE is written in Java, it throws this exception when it attempts to contact CN2 on its heartbeat port (default

9050

) but the connection is rejected. This is why CN2 is marked as

Alive: false

in

SHOW COMPUTE NODES

. Here is why this is happening and how to fix it: 1. The CN process likely crashed or failed to start In your previous logs, you saw

run start_cn.sh

. If the process encountered a fatal error immediately after that, it would stop listening on port 9050. * Check the process: Run

kubectl exec starrocks-cn-2 -n trino -- ps aux | grep starrocks

. If you don't see a process named

starrocks_be

, the CN has crashed. * Check for fatal logs: Run:

bash
    kubectl exec starrocks-cn-2 -n trino -- cat /opt/starrocks/cn/log/cn.INFO | grep -i "fail\|fatal\|error"

Look for errors like "failed to init storage engine" or "port already in use". 2. IP/Network Configuration (

priority_networks

) If CN2 is running but the FE still can't connect, it might be because the CN is listening on the wrong network interface (e.g.,

127.0.0.1

instead of the Pod IP). * Check the

cn.conf

inside the pod:

bash
    kubectl exec starrocks-cn-2 -n trino -- cat /opt/starrocks/cn/conf/cn.conf | grep priority_networks

* In Kubernetes,

priority_networks

should typically be set to the CIDR of your Pod network (e.g.,

10.x.x.0/24

). If it's missing or wrong, the CN might register with an IP the FE cannot reach. 3. Verification of the Registration In your MySQL client, check the registered address for CN2:

sql
SHOW COMPUTE NODES\G

* Look at the

IP

and

HeartbeatPort

columns for CN2. * Try to

ping

or

telnet

that IP/Port from an FE pod to see if it's reachable. 4. Common Kubernetes Issue: Port Conflict If you are running multiple StarRocks pods on the same physical K8s node without using a proper CNI (like Calico/Flannel), ensure that the ports (9050, 9060, 8040, etc.) aren't conflicting. However, this is rare in standard K8s setups. Recommendation The most likely cause is that the CN process failed a few seconds after

start_cn.sh

was called. Please check the

<http://cn.INFO|cn.INFO>

or

cn.WARNING

logs inside the CN2 pod to see the final error message before the process stopped.

bash
kubectl exec starrocks-cn-2 -n trino -- tail -n 50 /opt/starrocks/cn/log/cn.INFO

References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • administration/management/logs.md • deployment/deploy_manually.md • faq/Deploy_faq.md • administration/management/BE_configuration.md References • administration/management/logs.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • administration/Meta_recovery.md

@Rocky, this is changing, now CN-0 is not alive and all the starrocks pods are restarting even FE went down

If all StarRocks pods, including the FE, are now restarting, your cluster is likely in a cascading failure or a leader election loop. When the FE (FrontEnd) goes down or restarts, the entire cluster becomes unstable because the CN/BE nodes lose their coordinator. Follow these steps immediately to find the root cause: 1. Identify the Restart Reason (K8s level) Check why Kubernetes is killing the pods. Run this for the FE and one of the CNs:

bash
kubectl describe pod <fe-pod-name> -n trino
kubectl describe pod starrocks-cn-0 -n trino

Look at the

Last State

and

Reason

fields: *

OOMKilled

: The pod exceeded its memory limit. This is common if the FE JVM heap or the CN process is under-resourced. *

Error (Exit Code 1 or 137)

: 137 usually means OOM; 1 means a process crash. *

Readiness/Liveness probe failed

: The pod was killed because it didn't respond to health checks in time. 2. Check the FE Logs (The "Why") Since the FE is restarting, you need to check the logs from the previous failed instance:

bash
kubectl logs <fe-pod-name> -n trino --previous | tail -n 100

Look for these specific error patterns: *

java.lang.OutOfMemoryError: Java heap space

: Your FE needs more memory (

-Xmx

in

fe.conf

). *

transfer FE type from LEADER to UNKNOWN. exit

: This means the FE lost its leadership (likely due to a long "Full GC" pause or network lag) and shut itself down. *

com.sleepycat.je.rep.InsufficientReplicasException

: The FE cluster cannot form a quorum (needs at least 2 out of 3 FEs alive if you have a 3-FE setup). *

Clock delta: XXX ms ... exceeds max permissible delta

: The system clocks on your K8s nodes are out of sync. 3. Check for Resource Starvation If multiple pods are failing at once, the physical Kubernetes nodes might be overloaded.

bash
kubectl top nodes
kubectl top pods -n trino

If memory usage is near 90-100%, StarRocks processes will crash or trigger Full GCs that look like "not alive" states. 4. How to stabilize the cluster If the pods are in a

CrashLoopBackOff

, the easiest way to debug is to scale down and start one by one: 1. Scale down the CNs to 0:

kubectl scale statefulset <cn-name> -n trino --replicas=0

2. Focus on the FE: Ensure the FE (FrontEnd) is stable and shows

role: LEADER

. * Check with:

kubectl logs <fe-pod> -n trino

* If you can reach the MySQL port, run

SHOW FRONTENDS;

. 3. Scale CNs back up one by one: Once the FE is stable, bring up

CN-0

, then

CN-1

, then

CN-2

. Common Root Cause in K8s: If you haven't configured

priority_networks

in

fe.conf

and

cn.conf

, the pods might be trying to communicate on the wrong network interface after a restart, causing them to fail heartbeats and restart again. Can you share the last 20 lines of

kubectl logs <fe-pod> --previous

? That will pinpoint if it's a memory issue or a metadata/quorum issue. References • administration/Meta_recovery.md • faq/Deploy_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/logs.md • deployment/sr_operator.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md References • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_memory_problems.md