@Rocky why does initpwd container get the following error on eks:

ERROR 2005 (HY000): Unknown MySQL server host 'xxxx' (-3)
error code: 2005
Other unsuccessful errors, abnormal exit

The error

ERROR 2005 (HY000): Unknown MySQL server host 'xxxx' (-3)

in the

initpwd

container (usually part of the

kube-starrocks-initpwd

pod) is typically a transient DNS resolution error that occurs during the initial deployment of StarRocks on EKS. Why this happens The

initpwd

container is a helper pod (or job) responsible for connecting to the StarRocks Frontends (FE) to set the initial

root

password. According to the StarRocks documentation: 1. Expected Startup Behavior: The

kube-starrocks-initpwd

pod is designed to start immediately. However, it often starts before the FE pods are fully healthy or before the Kubernetes Service (DNS) for the FEs has been registered in CoreDNS. 2. DNS Failure: Because the hostname (e.g.,

<cluster-name>-fe-service

) cannot yet be resolved by the EKS cluster's DNS, the MySQL client inside the container throws

ERROR 2005

. 3. Automatic Retries: Kubernetes will automatically restart this pod (it will enter

Error

or

CrashLoopBackOff

states). This is expected behavior. What you should do * Wait: In most cases, you should simply wait. Once the FE pods are running and the service is discoverable, the next retry of the

initpwd

pod will succeed. * Check FE Status: Ensure your FE pods are actually reaching a

Running

and

Ready

state.

bash
    kubectl get pods -l "<http://starrocks.com/table-type=fe|starrocks.com/table-type=fe>"

* Verify Completion: The system is considered ready when all other pods show

1/1

and the

kube-starrocks-initpwd

pod shows a status of

Completed

. When to investigate further If the

initpwd

pod stays in

CrashLoopBackOff

for more than 10–15 minutes after the FEs are healthy, check for these issues: 1. CoreDNS Issues: Check if CoreDNS in your EKS cluster is functioning correctly (

kubectl get pods -n kube-system -l k8s-app=kube-dns

). 2. Custom Service Names: If you have customized the service names in your Helm

values.yaml

, ensure the

initpwd

container is trying to connect to the correct hostname. 3. Security Groups/Network Policies: Ensure that pods in your EKS cluster can communicate with each other on the MySQL query port (default

9030

) and that they can reach the CoreDNS service. References • faq/Others.md • quick_start/helm.md • deployment/post_deployment_setup.md • deployment/deploy_manually.md • sql-reference/Error_code.md

@Rocky all cn replicas and all fe replica but the first fe replica keep restarting

This pattern—where only the first FE (Frontend) is stable while all other FEs and CNs (Compute Nodes) are in a restart loop—usually points to a connectivity or registration issue between the nodes. In a StarRocks cluster, the follower FEs and all CNs must be able to communicate with the leader FE to join the cluster. If they cannot, they will fail to start and enter a

CrashLoopBackOff

. 1. The "No Quorum" Trap If you have configured 3 FE replicas, StarRocks requires a majority (at least 2) to elect a leader and become functional. * If FE-1 and FE-2 cannot talk to FE-0, FE-0 will never become the "Leader." * Without a Leader, the CNs cannot register and will also crash. * Action: Check the logs of FE-1 or FE-2. Look for

UnknownMasterException

or

failed to connect to helper

. 2. Common Causes in EKS A. Incorrect

priority_networks

This is the most frequent cause in Kubernetes. If a pod has multiple network interfaces, StarRocks might pick the wrong IP address (e.g., a Docker bridge IP instead of the EKS pod IP). * Symptom: FE-1 tries to join FE-0 using an IP that FE-0 doesn't recognize or can't reach. * Fix: Ensure

priority_networks

is set in your

values.yaml

(Helm) or StarRocksCluster spec (Operator) to match your VPC pod CIDR (e.g.,

10.0.0.0/16

). B. Port Connectivity (Security Groups / Network Policies) Even if pods are in the same cluster, EKS Security Groups or Kubernetes NetworkPolicies might block internal ports. * FE to FE communication: Needs port

9010

(Edit log port) open between all FE pods. * CN to FE communication: Needs port

9050

(FE Heartbeat port) and

9030

(FE Query port) open. * Action: Verify that your EKS node security group allows traffic on these ports within the VPC. C. DNS Resolution Failures Follower FEs and CNs use the FE service name (e.g.,

starrocks-fe-0.starrocks-fe-search

) to find the leader. * Action: Run a debug pod and try to

ping

or

telnet

the FE service address from within the cluster. 3. How to Diagnose To find the exact reason, you must look at the logs of the failing pods: 1. Check FE-1 logs:

bash
    kubectl logs <fe-1-pod-name>

Look for:

com.sleepycat.je.rep.UnknownMasterException

or

fail to get master

. This confirms it can't find FE-0. 2. Check CN logs:

bash
    kubectl logs <cn-pod-name>

Look for:

fail to find master

or

heartbeat timeout

. 3. Check FE-0 status: Connect to the stable FE-0 using MySQL and run:

sql
    SHOW FRONTENDS;
    SHOW COMPUTE NODES;

If

Alive

is

false

for the other nodes, the leader hasn't received their heartbeats. Summary Checklist * [ ] Is

priority_networks

configured correctly for the EKS pod network? * [ ] Is port

9010

open for FE-to-FE communication? * [ ] Is port

9050

open for CN-to-FE heartbeats? * [ ] Does the FE-1 log show a connection timeout to FE-0? References • quick_start/helm.md • faq/Deploy_faq.md • administration/Meta_recovery.md