@Rocky I am running starrocks in kubernetes, one of the fe pod is unable to comeup and I do not see any informative pod logs that is useful, since the pod is in crashloop back I am unable to connect to container and check the fe logs, however my other 2 fe pods are healthy, how can I see what is the issue with the 3rd pod .?

When one FE pod is in a

CrashLoopBackOff

and

kubectl logs

is not providing enough information, it is often because the StarRocks FE process is writing its logs to files on disk rather than to the standard output (stdout). Here are the steps to troubleshoot and find the error: 1. Check the previous container logs Sometimes the error occurs just before the container restarts. You can view the logs from the previous failed instance:

bash
kubectl logs <failed-fe-pod-name> --previous

2. Check Kubernetes events Check if the pod is being killed by Kubernetes due to Resource Limits (OOMKilled) or node issues:

bash
kubectl describe pod <failed-fe-pod-name>

Look for the Events section at the bottom and the Last State under Containers. If it says

OOMKilled

, you need to increase the memory limits in your StarRocksCluster spec. 3. Access logs from the Persistent Volume (PV) If

kubectl logs

is empty, it means the logs are being written to the

fe/log

directory inside the container's persistent storage. Since the pod is crashing, you cannot

exec

into it. You have two options: Option A: Create a Temporary Debug Pod You can spin up a simple Ubuntu/BusyBox pod that mounts the same PersistentVolumeClaim (PVC) as the failing FE pod to read the files directly. 1. Identify the log PVC for the failing pod (e.g.,

fe-log-starrocks-fe-2

). 2. Create a `debug-pod.yaml`:

yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: fe-debug
   spec:
     containers:
     • name: debug
       image: busybox
       command: ["sleep", "3600"]
       volumeMounts:
       • name: log-dir
         mountPath: /log
     volumes:
     • name: log-dir
       persistentVolumeClaim:
         claimName: <your-fe-log-pvc-name>

3. Run

kubectl apply -f debug-pod.yaml

, then

kubectl exec -it fe-debug -- ls -rt /log

to find and

cat

the latest

fe.warn.log

or

fe.out

. Option B: Redirect logs to Console (for future prevention) If your StarRocks version is 3.2.0 or later, you can ensure logs appear in

kubectl logs

by setting the environment variable

SYS_LOG_TO_CONSOLE=1

in your FE spec. 4. Check

fe.out

Often, the most informative log for a crash is

fe.out

. This file captures JVM startup errors (like "insufficient memory for the Java Runtime Environment") and stack traces that occur before the main logging system (

fe.log

) even starts. Use the "Debug Pod" method above to check this file specifically. 5. Check the cluster state from healthy pods Log into one of the healthy FE pods and check how the cluster perceives the 3rd node:

sql
-- Connect via mysql client to a healthy FE
SHOW FRONTENDS;

* Is the 3rd FE listed? If not, it failed to join the group. * Is

Join

true or false? If false, there might be a network connectivity issue or a metadata mismatch. Common causes for FE CrashLoop in K8s: * Insufficient Memory: The FE JVM couldn't start. Check

fe.out

for JVM errors. * Metadata Inconsistency: If the pod's IP or hostname changed and it doesn't match the metadata recorded in the Leader FE, it may fail to join. * Disk Full: Check if the log or metadata PVC is 100% full. References • quick_start/helm.md • deployment/sr_operator.md • deployment/helm.md

@Rocky I was able to run the starrocks startup script within the. container, however I do not see the starrocks process is up and running and I do no see any logs in fe.log or fe.out,

If you are running the

start_fe.sh

script manually and the process exits immediately without writing any logs to

fe.log

or

fe.out

, it usually indicates a failure at the JVM initialization or shell script level before the StarRocks logging system can even start. Since your other two FEs are healthy, follow these steps to diagnose the specific issue with the 3rd pod: 1. Run the startup script in the Foreground By default, most people run

./bin/start_fe.sh --daemon

. This sends the process to the background. if the JVM crashes immediately, the error might not be captured in the file. Try running without

--daemon

to see the error message directly on your terminal:

bash
# Inside the container shell
./bin/start_fe.sh

If there is a Java environment issue, a syntax error in

fe.conf

, or an invalid JVM memory flag, it will print to your console now. 2. Check for a locked PID file If a previous process crashed and left a

fe.pid

file, the new process might fail to start because it thinks another instance is already running. * Check for the file:

ls -l fe/bin/fe.pid

(or wherever your

PID_DIR

is set). * If it exists and you are sure no FE process is running, delete it:

rm fe/bin/fe.pid

. 3. Verify Disk Space and Permissions If the Persistent Volume (PV) assigned to this 3rd pod is full, the JVM cannot write logs or metadata, which can cause an immediate exit.

bash
df -h                 # Check disk space
ls -ld fe/log fe/meta # Check if the user has write permissions

4. Check for OOMKilled (Out of Memory) Even if the pod logs are empty, Kubernetes tracks why the container died.

bash
kubectl describe pod <failed-fe-pod-name>

Look for

Last State: Terminated

and the

Reason

. If it says

OOMKilled

, the container's memory limit is too low for the Java Heap (

-Xmx

) you have configured in

fe.conf

. _ _Rule of thumb:* Container memory limit should be at least 10-20% higher than the JVM

-Xmx

setting to account for off-heap memory. 5. Potential Metadata Corruption If the 3rd FE crashed during a metadata write, its local

BDBJE

(Berkeley DB) state might be corrupted. Warning: Only do this if you have a healthy Leader and this is a non-leader node. 1. Scale the FE replicas to 2 (to stop the crashing pod). 2. Delete the PVC associated with the 3rd FE's metadata (e.g.,

fe-meta-starrocks-fe-2

). 3. Scale back to 3. 4. The StarRocks Operator will provision a new PVC, and the FE will join as a new node and synchronize metadata from the Leader. 6. Verify Java Environment Ensure the script can actually find Java:

bash
java -version
echo $JAVA_HOME

If you are using a custom image, the path to Java might be missing or incorrect in the shell's environment. 7. Check connectivity to the Leader The 3rd FE must be able to reach the Leader FE on the

edit_log_port

(default 9010). Use

telnet

or

nc

from within the 3rd pod's container (if you can keep it alive long enough with a

sleep

command) to check the other FEs:

bash
nc -zv <leader-fe-service-ip> 9010

Recommended First Step: Use

kubectl edit

to change the pod's command to

sleep 3600

. This will keep the pod "Running" so you can

exec

into it, inspect the files, and try running

./bin/start_fe.sh

manually in the foreground as suggested in the foreground as suggested in Step 1. References • administration/management/logs.md • administration/management/FE_configuration.md • deployment/deploy_manually.md

@Rocky I was able to start the script manually and I do not see any error, however I do not see any disk sape issue or memory issue and I do not see any logs in fe.log and fe.out, I arleady have setting enabled to write the logs to log file.

If you are running the

start_fe.sh

script manually, it reports no errors, yet the process is not running and the log files are empty, you are likely hitting a "silent failure" in the shell script or the JVM before Log4j can initialize. Here are the specific things to check in this scenario: 1. Check for the "Double Backgrounding" trap If you run

./bin/start_fe.sh --daemon

, the script starts the FE and then exits. If the FE process then crashes 0.5 seconds later, the script won't show an error. Action: Run it in the foreground and keep the terminal open:

bash
./bin/start_fe.sh

If it returns to the command prompt immediately without any output, the script itself is likely failing to find a dependency (like Java) or failing to create a directory. 2. Check the "Ulimit" or "File Descriptor" limit The FE requires a high number of file descriptors (usually 65536). In some K8s environments, if the

ulimit

is too low, the JVM may exit immediately. Action: Check your limits inside the container:

bash
ulimit -n

If it's low (e.g., 1024), you need to set the

securityContext

in your K8s deployment to allow higher limits. 3. Verify

STARROCKS_HOME

and

sys_log_dir

If

sys_log_dir

in

fe.conf

points to a location that the current user cannot write to, or if the directory doesn't exist, the FE might fail to initialize its logging system. Action: 1. Check

fe.conf

for the

sys_log_dir

setting. 2. Verify you can manually create a file in that directory:

touch /path/to/your/log/test_file

. 4. Search for a "Hidden"

fe.out

Sometimes the script redirects output to a different location than you expect. Action: Run a search for any file modified in the last 5 minutes:

bash
find / -mmin -5 -type f

This might reveal that logs are being written to a temporary directory or a different path defined in your environment variables. 5. Check for

fe.pid

If a

fe.pid

file exists in the

bin

or

run

directory from a previous failed run, the script might think the FE is already running and simply do nothing. Action:

bash
ls -l bin/*.pid
# If a pid file exists, delete it and try starting again.

6. Java Version Mismatch If you have multiple Java versions, the script might be picking up an incompatible one. StarRocks FE generally requires Java 8 or 11 (depending on the version). Action:

bash
java -version

If it returns nothing or an error, you must set

JAVA_HOME

in

fe.conf

. 7. Inspect the script with

bash -x

If the script is exiting silently, you can "trace" exactly what it is doing:

bash
bash -x ./bin/start_fe.sh

This will print every line of the script as it executes. Look for the last few lines before it exits to see if it's failing on a specific check or path. Summary of most likely cause: Since you see nothing in

fe.out

, it is almost certain that the JVM is failing to start (likely due to memory flags or missing Java) or the shell script is exiting because a required directory/file is missing. Running with

bash -x

is the fastest way to see why the script is stopping. References • deployment/deploy_manually.md • administration/management/logs.md