@Rocky Starrocks 3.5.15 in shared data mode deployed through kubernetes operator. I've updated sidecar container code - two fe nodes started normally, but third one keeps failing. No special errors in the report. There are resources available on the cluster. What can I do to debug?

For a StarRocks 3.5.15 cluster in shared-data mode on Kubernetes, where two FE nodes are healthy but the third is failing after a sidecar update, the issue is likely related to the sidecar's health, pod-level orchestration, or the FE's ability to join the metadata (BDBJE) group. Here is a step-by-step guide to debugging this: 1. Check Container and Pod Status Since you updated sidecar code, the first place to look is the Kubernetes status. A pod might stay in

Running

but never reach

Ready

if the sidecar fails its liveness/readiness probes. * Check Pod Events:

bash
    kubectl describe pod <failing-fe-pod-name>

Look at the Events section for

Unhealthy

,

Back-off restarting failed container

, or probe failures. * Check Sidecar Logs:

bash
    kubectl logs <failing-fe-pod-name> -c <sidecar-container-name>

Look for runtime errors, permission issues, or networking timeouts in your new code. 2. Inspect FE-Specific Logs If the sidecar is running but the FE process itself is failing, check the FE logs located inside the container (usually

/opt/starrocks/fe/log

): *

fe.out

: This captures standard output/error. If the JVM fails to start or there is a fatal startup error, it will be here. *

fe.warn.log

: Look for "BDBJE" errors or "UnknownMasterException."

bash
    kubectl logs <failing-fe-pod-name> -c starrocks-fe

3. Verify Cluster Membership Connect to one of the healthy FE nodes via MySQL and check the cluster state:

sql
SHOW FRONTENDS;

* Check if the failing node is listed. * Check the

IsAlive

and

Role

columns. If it shows

IsAlive: false

, the cluster cannot communicate with it. * Ensure the

Host

(IP/FQDN) matches the current pod's network identity. In Kubernetes, if the pod's IP changed and the metadata wasn't updated, the node might fail to join. 4. Shared-Data Metadata Sync (BDBJE) In shared-data mode, StarRocks still uses BDBJE for FE metadata replication. If the third node's metadata directory is corrupted or out of sync: 1. Check for Locks: Ensure the Persistent Volume (PV) isn't stuck or mounted with incorrect permissions. 2. Clean Metadata (Last Resort): If you see

ReplicaWriteException

or

HandshakeError

in

fe.warn.log

, the node's local metadata might be inconsistent. * You can try dropping the failed follower:

ALTER SYSTEM DROP FOLLOWER "host:port";

* Delete the data in the FE's

meta

directory (the PVC content). * Re-add it:

ALTER SYSTEM ADD FOLLOWER "host:port";

* The node should then perform a "Remote Bootstrap" from the Leader. 5. Resource and Networking Checks * Sidecar Impact: Does your sidecar consume significant memory or CPU? Even if the cluster has resources, the Pod's Resource Limits (if defined in the CRD) might be hit, causing the OOM Killer to target a container. * Edit Log Port: Ensure the sidecar change didn't accidentally block or interfere with port

9010

(default

edit_log_port

), which FE nodes use to talk to each other. 6. StarRocks Operator Specifics Check the Operator's own logs to see if it's failing to reconcile the third pod:

bash
kubectl logs -n starrocks-operator-system Labeled starrocks-operator

The operator might be stuck in a loop trying to update the pod if the sidecar configuration is considered "invalid" by the operator's internal logic. References • administration/Meta_recovery.md