< Rocky> how to recover starrocks metadata fe we were runnin StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> how to recover starrocks metadata(f...

Abhishek Yadav

04/13/2026, 11:14 AM

@Rocky how to recover starrocks metadata(fe), we were running with 5 pod with pvc attached, for 1 pod pvc deleted..now after login using mysql seeing no database

Rocky

04/13/2026, 11:15 AM

It sounds like your StarRocks cluster has entered a "split-brain" or "invalid leader" state because the pod with the deleted PVC started with a clean metadata state and potentially became the leader or caused the other nodes to sync with its empty state. In a 5-node FE cluster, the metadata is replicated using BDB JE. If one node's PVC is deleted, it becomes a "fresh" node. If the other 4 nodes were still healthy, they should have maintained the data. However, if the empty node was elected leader (or the service is routing you to it), you will see no databases. ⚠️ Warning: Metadata recovery is sensitive. Before proceeding, back up the
meta/
directory of all remaining FE pods that still have their PVCs. Step 1: Verify the Cluster Status Log in to the FE via MySQL and run:

Copy code

sql
SHOW FRONTENDS;

Check which node is the LEADER and whether the other nodes are

Alive

. • If the Leader is the pod that lost its PVC, it explains why you see no databases. • If a different node is Leader but you still see no databases, the metadata may have been corrupted or overwritten during a re-sync. Step 2: Identify the Pod with the Latest Metadata You need to find which of the remaining 4 pods has the most recent "good" metadata. 1. Scale your FE StatefulSet to

to stop all nodes and prevent further corruption. 2. For each of the 4 healthy PVCs, you can use the BDB JE utility to find the latest VLSN (Version Log Sequence Number). You may need to run a temporary pod and mount the PVC to do this.

Copy code

bash
   # Run this inside the FE container lib directory
   java -jar fe/lib/starrocks-bdb-je-*.jar DbPrintLog -h /path/to/your/meta/bdb/ -vd | grep lastVLSN

3. The node with the largest
lastVLSN
is your best candidate for recovery. Step 3: Perform Metadata Recovery (Last Resort) Follow these steps to force the node with data to become the leader: 1. Prepare the "Best" Pod: Choose the pod identified in Step 2. Modify its

fe.conf

(via ConfigMap or by mounting the volume) to add this line:

Copy code

properties
   bdbje_reset_election_group = true

2. Start ONLY this one Pod: Scale the FE StatefulSet to

(ensuring it's the specific pod you chose, or manually start it). 3. Verify Data: Log in via MySQL. You should now see your databases. Run

SHOW FRONTENDS;

to confirm this node is the

LEADER

. 4. Rejoin the Other Nodes: • For the other 4 pods, delete their
meta/
directory entirely. This ensures they don't try to use old/corrupt metadata. • Scale the FE StatefulSet back to

. • The other pods will start, see the Leader, and automatically download the correct metadata image. 5. Clean Up: Once the cluster is healthy and all 5 nodes are

Alive

in `SHOW FRONTENDS;`: • Remove

bdbje_reset_election_group = true

from your

fe.conf

. • Restart the Leader pod one last time to return to a normal state. Summary Checklist • ☐ Scale FE to 0 immediately to stop sync issues. • ☐ Back up the

meta/

directories of the 4 healthy pods. • ☐ Identify the pod with the highest VLSN. • ☐ Use bdbje_reset_election_group = true
on that pod only. • ☐ Wipe
meta/
on the other 4 pods before restarting them. If you have a recent metadata image backup (usually found in

meta/image/

), you can also use that to restore a node from scratch. If you still see no databases after forcing a leader, please check if your

fe.log

contains any "checkpoint" or "image" errors. References • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 12:51 PM

we still have vol available, created new vpc to point older vol... now when I am trying to scale fe pod to 1 its restarting continously @Rocky

Rocky

04/13/2026, 12:51 PM

The continuous restarting of your FE pod (CrashLoopBackOff) is likely happening because the FE is trying to start using the recovered metadata from a 5-node cluster, but since it's the only node running, it cannot reach a quorum (the majority required for BDB JE to elect a leader). In a 5-node cluster, at least 3 nodes must be alive. Additionally, if the pod's IP address has changed since the PVC was last used, the metadata will mismatch the current environment. Follow these steps to force this single pod to start and recover your databases: Step 1: Check the Logs First, confirm the exact error. Look for "Election failed" or "RepGroupDB" errors:

Copy code

bash
kubectl logs <fe-pod-name>
# or if it just crashed
kubectl logs <fe-pod-name> --previous

Step 2: Force the FE to start as a Single-Node Leader To fix the quorum issue and the "no database" view, you must force this FE to reset its replication group and become the Leader. 1. Modify
fe.conf
: Add the following parameter to your FE configuration (via your ConfigMap or by editing the file if you have access):

Copy code

properties
    bdbje_reset_election_group = true

_Note: If you are on a very old version of StarRocks, use

metadata_failure_recovery = true

instead._ 2. Scale to 1: Ensure your FE StatefulSet is scaled to

so only the pod with the recovered volume is running.

Copy code

bash
    kubectl scale sts <fe-sts-name> --replicas=1

3. Wait for Startup: The FE should now start successfully. It will ignore the other 4 missing nodes, reset the cluster information to itself only, and become the LEADER. 4. Verify Databases: Log in via MySQL:

Copy code

sql
    SHOW DATABASES;
    SHOW FRONTENDS;

You should now see your old databases. In

SHOW FRONTENDS

, you will likely only see this one node as

Alive

. Step 3: Re-integrating the other 4 Pods Once you have verified your data is back: 1. Remove the recovery flag: Delete

bdbje_reset_election_group = true

from your

fe.conf

. Do not skip this step, or the node will reset the group every time it restarts. 2. Restart the Leader: Restart the current FE pod once more to ensure it runs in "normal" mode. 3. Wipe and Re-add Followers: For the other 4 pods (the ones that lost their PVCs or have empty volumes): * Ensure their

meta/

directories are completely empty. * Scale the FE StatefulSet back to

. * They will join as new nodes and synchronize the metadata automatically from the Leader. * If they don't appear in

SHOW FRONTENDS

, you may need to manually run

ALTER SYSTEM DROP FOLLOWER "host:port"

and then

ALTER SYSTEM ADD FOLLOWER "host:port"

for the specific IPs of the new pods. Troubleshooting IP Changes If the pod still restarts after adding the reset flag, it might be due to an IP mismatch. Ensure your

fe.conf

has

priority_networks

configured correctly to match the current Kubernetes pod network (e.g.,

10.x.x.0/24

), so the FE binds to the correct interface. References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 12:58 PM

@Rocky container still getting restarted --nothing in log.

Rocky

04/13/2026, 12:58 PM

If the container is restarting and you see nothing in the logs (i.e.,

fe.log

and

fe.warn.log

are empty or not updating), it typically means the process is failing before the logging framework can even initialize. In a Kubernetes environment where you have reattached an old volume to a new PVC, the most common culprits are: 1. File Permission Mismatch The most likely reason for "no logs" and immediate crashes is that the FE process does not have permission to write to the reattached volume. * The Issue: The files in the "older volume" are likely owned by a UID/GID from the previous installation. If the new pod is running with a different user, it will fail to open the metadata or log files. * The Fix: You can add an

initContainer

to your FE StatefulSet to fix the permissions:

Copy code

yaml
    initContainers:
    • name: volume-permissions
      image: busybox
      command: ["sh", "-c", "chown -R 1000:1000 /opt/starrocks/fe/meta /opt/starrocks/fe/log"] # Adjust UID 1000 to match your StarRocks user
      volumeMounts:
      • name: fe-meta
        mountPath: /opt/starrocks/fe/meta
      • name: fe-log
        mountPath: /opt/starrocks/fe/log

2. Check
fe.out
(The "Hidden" Log) StarRocks writes early startup errors (JVM failures, class errors, or configuration syntax issues) to

fe.out

. Since

kubectl logs

usually only streams stdout, if the startup script redirects stdout to a file, you won't see it in

kubectl

. * Action: Try to read the file directly if the pod stays up for a few seconds:

Copy code

bash
    kubectl exec <fe-pod-name> -- cat fe/log/fe.out

If it crashes too fast, use a temporary "debug" pod to mount that same PVC and inspect the contents of

fe/log/fe.out

. 3. Kubernetes Liveness/Readiness Probes If your metadata is large, BDB JE may take a long time to "reorganize" or "replay" logs after a recovery flag is set. * The Issue: Kubernetes might think the pod is "Unhealthy" and kill it before it finishes starting. * The Fix: Increase

initialDelaySeconds

and

failureThreshold

for your liveness and readiness probes in the StatefulSet spec (e.g., set

initialDelaySeconds: 300

). 4. JVM Heap vs. Container Memory If you manually configured

-Xmx

fe.conf

FE_OPTS

to be close to the container's memory limit, the pod might be getting OOMKilled by the kernel before Java even starts. * Action: Check the output of

kubectl describe pod <pod-name>

. Look for

Reason: OOMKilled

Exit Code: 137

. 5. IP Address Mismatch in Metadata Since you are scaling a 5-node cluster down to 1 and reattaching a volume, the metadata inside that volume still thinks it belongs to a specific IP/Host. * Action: Ensure you are using

bdbje_reset_election_group = true

fe.conf

. * Action: Check your

priority_networks

fe.conf

. If the new Pod's IP doesn't match the CIDR defined there, the FE might fail to start because it cannot find a valid local IP to bind to. Recommended Next Steps: 1. Describe the pod:

kubectl describe pod <fe-pod-name>

to see the exit code and events. 2. Check Previous Logs:

kubectl logs <fe-pod-name> --previous

might catch the last words of the process. 3. Check Permissions: Ensure the

meta

and

log

directories are writable by the container user. 4. Confirm Flag: Ensure you only have

bdbje_reset_election_group = true

and NOT

metadata_failure_recovery = true

(which is for much older versions and can cause

DatabaseNotFoundException

). References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Rocky

04/13/2026, 12:58 PM

If the container is restarting continuously and

fe.log

is empty, it means the process is crashing before the StarRocks logging system can even initialize. This is usually caused by environment issues, JVM failures, or permission problems. Please check the following in order: 1. Check
kubectl describe pod
Run this to see the Exit Code and Reason:

Copy code

bash
kubectl describe pod <fe-pod-name>

• Exit Code 137: OOMKilled (The pod needs more memory). • Exit Code 1: General error (often a typo in

fe.conf

or permission denied). • Exit Code 127: Command not found (unlikely here). 2. Check Permissions on the Metadata Volume Since you attached an "older vol" to a "new PVC," the file ownership might be wrong. If the StarRocks process (usually user

starrocks

root

) cannot write to the volume, it will crash immediately. • Try running a temporary "debug" pod mounting that same PVC and check if the files are owned by the correct user. • If you have access to the node or a shell, ensure the

meta/

directory and all its contents are writable:

Copy code

bash
  # Inside the pod or via a helper pod
  chown -R starrocks:starrocks /opt/starrocks/fe/meta
  chmod -R 755 /opt/starrocks/fe/meta

3. Check for a Corrupt
fe.conf
If you added

bdbje_reset_election_group = true

, ensure there are no typos, hidden characters, or duplicate entries in the configuration file. If the parser fails, the FE won't start. • Ensure the line is exactly:

bdbje_reset_election_group = true

• Check if you have any other conflicting recovery flags like

metadata_failure_recovery

. 4. Verify the
fe/meta/image/ROLE
file Since you are trying to recover from a specific volume, look at the

ROLE

file in your metadata directory:

Copy code

bash
cat /path/to/meta/image/ROLE

• If the file says

role=OBSERVER

, the

bdbje_reset_election_group

flag might not work directly. You must manually edit this file and change it to

role=FOLLOWER

before starting the pod with the reset flag. 5. Check JVM and Memory StarRocks FE requires a significant amount of JVM heap. • Check the

JAVA_OPTS

in your

fe.conf

or environment variables. • If your Kubernetes memory limit is (for example) 8GB, but your JVM is configured for

-Xmx8g

, the pod will crash immediately because of overhead. Ensure the pod limit is at least 10-20% higher than the JVM heap. 6. "Ghost" Processes If the PVC was not cleanly unmounted or if the previous pod didn't shut down gracefully, there might be a

java.pid

or a lock file in the

meta/bdb

directory. • Check

fe/meta/bdb/

for any files ending in

.lock

and try removing them (after making a backup). Recommended Next Step: Try to get any output by running the start command manually. If you can exec into the pod (if it stays up long enough) or use a

command: ["sleep", "3600"]

override in the YAML, run:

Copy code

bash
./bin/start_fe.sh --daemon
# Then immediately check:
cat log/fe.out

The

fe.out

file often contains the "true" error (like a Java StackTrace) that doesn't make it into

fe.log

Abhishek Yadav

04/13/2026, 12:59 PM

@Rocky can not exec to pod as its continously restarting.. ReasonReason Error - exit code: 255

Rocky

04/13/2026, 12:59 PM

Since your pod is crashing before it can even write to

fe.log

, the standard StarRocks logs won't help. You need to "pause" the pod to inspect it manually. Follow these steps to stop the restart loop and find the root cause: Step 1: Force the Pod to stay "Running" Modify your FE StatefulSet/Deployment to override the startup command. This prevents the FE from trying to start (and crashing) and allows you to

exec

into the pod. 1. Edit the StatefulSet:

Copy code

bash
    kubectl edit sts <your-fe-statefulset-name>

2. Add a
command
override to the FE container spec:

Copy code

yaml
    spec:
      containers:
      • name: starrocks-fe
        command: ["/bin/bash", "-c", "sleep 3600"] # This keeps the pod alive for 1 hour

3. Wait for the pod to reach
Running
state. Step 2: Manually trigger the FE and check
fe.out
Now that the pod is stable, enter the container and try to start the FE manually. This will reveal why it's failing. 1. Exec into the pod:

Copy code

bash
    kubectl exec -it <fe-pod-name> -- /bin/bash

2. Check file permissions: Ensure the user has rights to the meta directory. If you see

root

owning files while the process runs as

starrocks

, that is your problem.

Copy code

bash
    ls -al /opt/starrocks/fe/meta

3. Run the start script manually and check the output:

Copy code

bash
    cd /opt/starrocks/fe/
    ./bin/start_fe.sh --daemon

    # IMMEDIATELY check this file (this is where the real error is):
    cat log/fe.out

Step 3: Common Causes to Look For in
fe.out
Once you see the contents of

fe.out

, look for these specific errors: * "IP address mismatch": If you see an error about an IP not matching the one in the metadata, check the file

meta/image/ROLE

. If the IP there is different from the current Pod IP: * Add

priority_networks = <current_pod_subnet>/24

fe.conf

. * Ensure the IP in the

ROLE

file matches what StarRocks is trying to bind to. * "DatabaseNotFoundException: _jeRepGroupDB": This usually happens if you have

metadata_failure_recovery=true

fe.conf

when it's not needed. Remove it. * "Insufficient memory" / "OOM": The JVM cannot allocate the heap size specified in

JAVA_OPTS

. Check if your K8s memory limits are high enough. * "Lock file existing": If you see a BDB JE lock error, look in

meta/bdb/

and delete any file ending in

.lock

. Step 4: Verify the
ROLE
file If you are trying to recover metadata, open the

meta/image/ROLE

file:

Copy code

bash
cat /opt/starrocks/fe/meta/image/ROLE

* If
role=OBSERVER
: The recovery flag

bdbje_reset_election_group = true

will not work. You must manually change the text in that file to

role=FOLLOWER

first, then try starting again. Summary 1. Use

command: ["sleep", "3600"]

to stop the restart loop. 2.

exec

in and check log/fe.out
. 3. Fix the IP mismatch or permissions revealed in

fe.out

. 4. Remove the

command

override from your YAML once the manual start works. References • administration/Meta_recovery.md

Rocky

04/13/2026, 1:01 PM

The Exit Code 255 (which often maps to -1) indicates that the FE startup script or the JVM itself is crashing before it can even initialize the logging system. This is common when moving volumes between VPCs because of permission mismatches or network environment changes. Since you cannot

exec

due to the restart loop, you must "freeze" the pod to inspect it. Step 1: Force the Pod to stay alive If you are using a StatefulSet, you can patch it to run a

sleep

command instead of the StarRocks entrypoint. This stops the crash loop.

Copy code

bash
# Replace <sts-name> with your FE StatefulSet name
kubectl patch sts <sts-name> --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["/bin/bash", "-c", "sleep 3600"]}]'

Wait for the pod to restart. It will now stay in a

Running

state but won't start StarRocks. Now you can enter:

Copy code

bash
kubectl exec -it <fe-pod-name> -- /bin/bash

Step 2: Critical Checks inside the Pod Once inside the pod, check these three things in order: 1. File Permissions (Most Likely) When moving volumes across VPCs or storage providers, the UID/GID often changes. StarRocks usually runs as user

starrocks

(UID 1000). If the files are owned by

root

or a different ID, FE will exit with 255.

Copy code

bash
# Check ownership of the meta directory
ls -ln /opt/starrocks/fe/meta

# If the owner is not 1000, fix it:
chown -R 1000:1000 /opt/starrocks/fe/meta

2. Inspect
fe.out
This file contains the "raw" error that doesn't make it into

fe.log

Copy code

bash
# Try to start it manually once to see the error
/opt/starrocks/fe/bin/start_fe.sh --daemon
cat /opt/starrocks/fe/log/fe.out

* Look for:

java.net.BindException

(IP mismatch) or

Permission denied

. 3. Handle IP Mismatch (New VPC) If your new VPC uses a different IP range, the FE will crash because it sees its "old" IP in the metadata. 1. Check your current Pod IP:

hostname -i

. 2. Check the metadata role file:

cat /opt/starrocks/fe/meta/image/ROLE

. 3. If they don't match, you must add

priority_networks

to your

fe.conf

to match the new VPC subnet (e.g.,

priority_networks = 10.x.x.0/24

). Step 3: Recovering the Metadata Once permissions and IPs are fixed, you still need to address the fact that you have 1 pod trying to run a 5-node cluster's metadata. 1. While still in the "sleep" pod, edit `fe.conf`:

Copy code

properties
    # Add this line to force the FE to become leader of its own 1-node group
    metadata_failure_recovery = true

2. Start the FE manually:

./bin/start_fe.sh --daemon

. 3. Verify with

mysql

that you see your databases. 4. Crucial: Once it starts and you see your data, remove

metadata_failure_recovery = true

from

fe.conf

. 5. Remove the

kubectl patch

(set command back to null/default) to let the pod start normally. Why 255 specifically? * Java Memory: If your

JAVA_OPTS

(like

-Xmx

) is set higher than the Kubernetes RAM limit, the JVM returns 255. * Missing Binaries: If the move changed the environment and a script dependency is missing. * Config Syntax: A typo in

fe.conf

added during the VPC move. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • administration/management/logs.md • faq/operation_maintenance_faq.md

Abhishek Yadav

04/13/2026, 1:04 PM

Copy code

LOG_DIR = ${STARROCKS_HOME}/log
DATE = "$(date +%Y%m%d-%H%M%S)"
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx16384m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 9010
mysql_service_nio_enabled = true
sys_log_level = INFO

@Rocky is anything incorrect here ?

Rocky

04/13/2026, 1:04 PM

Yes, there are several issues in that snippet that are likely causing your Exit Code 255 and the empty logs. 1. The
DATE
variable and GC Log Path (Most Likely Cause) The line

DATE = "$(date +%Y%m%d-%H%M%S)"

is shell syntax. While the FE startup script (

start_fe.sh

) does "source" the config file, having complex shell commands inside

fe.conf

often causes issues with how the variables are exported to the JVM. * The Risk: If the

$DATE

variable is not correctly evaluated or exported, the JVM tries to start with a GC log path like

${LOG_DIR}/fe.gc.log.:time

. If the JVM cannot create or write to that specific filename, it will crash immediately with exit code 255 before any StarRocks code runs. * The Fix: Remove the custom

DATE

and

LOG_DIR

definitions from

fe.conf

. StarRocks handles these automatically. 2. Java Version vs. GC Log Syntax Your

JAVA_OPTS

uses the syntax

-Xlog:gc*...

, which is for Java 11 or higher. * If your environment is running Java 8, this syntax is invalid and the JVM will fail to start. (Java 8 uses

-Xloggc:<file>

). * Recommendation: Use the default StarRocks GC logging if possible, or ensure your Java version matches the syntax. 3. Memory Allocation (
-Xmx16384m
) You have set the heap to 16GB. * The Risk: In a containerized environment (Kubernetes), if your Pod's memory limit is also 16GB (or close to it, like 18GB), the container will be killed immediately (OOMKilled) or fail to initialize because the JVM needs extra memory for overhead (Off-heap, Metaspace, Stack). * The Fix: Ensure your Pod memory limit is at least 20-30% higher than your

-Xmx

setting. For a 16GB heap, your Pod limit should be ~20GB. 4. Redundant Variables

LOG_DIR

and

STARROCKS_HOME

are internally managed by the StarRocks startup scripts. Overriding them in

fe.conf

can lead to recursive definitions or path errors if the environment variables aren't set exactly as the script expects. ────────── Recommended "Clean"
fe.conf
Snippet Try replacing those lines with this standard version to see if it starts:

Copy code

properties
# Let the system handle LOG_DIR and DATE
# Simplified JAVA_OPTS (compatible with Java 11+)
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx16384m -XX:+UseG1GC"

# If you specifically need GC logs, use the standard format (Java 11+):
# JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx16384m -XX:+UseG1GC -Xlog:gc*:file=log/fe.gc.log:time,tags:filecount=10,filesize=50M"

http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 9010
mysql_service_nio_enabled = true
sys_log_level = INFO

How to verify: 1. Check Java Version: Run

java -version

inside the container (using the

sleep

trick mentioned earlier). 2. Test the Config: After updating

fe.conf

, try running

./bin/start_fe.sh --daemon

manually. 3. Check `fe.out`: If it still crashes, run

cat log/fe.out

. Any JVM startup error (like "Could not create the Java Virtual Machine") will be printed there. References • administration/Meta_recovery.md • faq/Deploy_faq.md • developers/development-environment/IDEA.md

Abhishek Yadav

04/13/2026, 1:12 PM

@Rocky

Copy code

root@kube-starrocks-fe-0:/opt/starrocks# ls -al /opt/starrocks/fe/meta
cat /opt/starrocks/fe/meta/image/ROLE
total 44
drwxr-xr-x. 5 root      root       4096 Jan 30 01:41 .
drwxr-xr-x. 1 starrocks starrocks    17 Apr 13 21:10 ..
drwxr-xr-x. 2 root      root      20480 Apr 13 16:48 bdb
drwxr-xr-x. 4 root      root       4096 Apr 13 18:24 image
drwx------. 2 root      root      16384 Jan 30 01:40 lost+found
#Mon Apr 13 18:24:24 CST 2026
role=FOLLOWER
hostType=
name=kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1776075864420
root@kube-starrocks-fe-0:/opt/starrocks#

Rocky

04/13/2026, 1:12 PM

The output you provided confirms the exact cause of your Exit Code 255. Problem 1: Permission Mismatch (The "Smoking Gun") Look at your

ls -al

output:

Copy code

text
drwxr-xr-x. 5 root      root       4096 Jan 30 01:41 .     <-- meta directory
drwxr-xr-x. 2 root      root      20480 Apr 13 16:48 bdb   <-- bdb directory
drwxr-xr-x. 4 root      root       4096 Apr 13 18:24 image <-- image directory

The StarRocks FE process runs as the starrocks
user, but your metadata files are owned by root
. When the FE tries to write a lock file or update the image, it gets a "Permission Denied" error at the OS level, causing the JVM to crash immediately with code 255. Problem 2: Role is FOLLOWER Your

ROLE

file says

role=FOLLOWER

. If you are trying to restore this as a single-node cluster, a Follower will never start because it is waiting for a Leader that doesn't exist. ────────── The Fix While you are still `exec`'d into the pod (using the sleep command), run the following: 1. Fix Permissions Change the owner of the entire metadata directory back to the

starrocks

user:

Copy code

bash
chown -R starrocks:starrocks /opt/starrocks/fe/meta

2. Force the FE to become Leader Since this node is currently a

FOLLOWER

, you need to force it to reset the election group so it can elect itself as the new Leader. Edit your

fe.conf

and add this line:

Copy code

properties
bdbje_reset_election_group = true

3. Test the startup manually Try starting the FE while still inside the pod to verify the fix:

Copy code

bash
# Switch to starrocks user if you are root
su starrocks 

# Start FE
/opt/starrocks/fe/bin/start_fe.sh --daemon

# Check the output file immediately for errors
cat /opt/starrocks/fe/log/fe.out

Next Steps 1. If it starts successfully: * Log in via MySQL (

query_port

9030). * Run

SHOW FRONTENDS;

to confirm this node is now the

LEADER

. * Crucial: Once it is running, remove

bdbje_reset_election_group = true

from your

fe.conf

and remove the

sleep

command from your Kubernetes YAML so it can start normally on the next reboot. 2. If it still fails: * Check

log/fe.out

again. Since permissions are fixed, it will now likely show a specific Java error or an IP mismatch error. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 1:20 PM

Copy code

Feb 04, 2026 6:43:15 PM com.baidu.jprotobuf.pbrpc.transport.RpcChannelFutureListener operationComplete
WARNING: build channel:[id: 0x4b7155c3, L:null ! R:/192.168.180.67:8060] failed
Feb 04, 2026 6:43:15 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPoolObjectFactory wrap
SEVERE: failed to get result from stp
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.180.67:8060
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)

Feb 04, 2026 6:43:15 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPool getChannel
SEVERE: Unable to validate object
java.util.NoSuchElementException: Unable to validate object
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
        at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:80)
        at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:73)
        at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:499)
        at jdk.proxy2/jdk.proxy2.$Proxy40.getTabletStats(Unknown Source)
        at com.starrocks.rpc.LakeServiceWithMetrics.getTabletStats(LakeServiceWithMetrics.java:102)
        at com.starrocks.catalog.TabletStatMgr$CollectTabletStatJob.sendTasks(TabletStatMgr.java:401)
        at com.starrocks.catalog.TabletStatMgr$CollectTabletStatJob.execute(TabletStatMgr.java:361)
        at com.starrocks.catalog.TabletStatMgr.updateLakeTableTabletStat(TabletStatMgr.java:313)
        at com.starrocks.catalog.TabletStatMgr.updateLakeTabletStat(TabletStatMgr.java:248)
        at com.starrocks.catalog.TabletStatMgr.runAfterCatalogReady(TabletStatMgr.java:112)
        at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72)
        at com.starrocks.common.util.Daemon.run(Daemon.java:98)

Feb 04, 2026 6:43:15 PM com.baidu.jprotobuf.pbrpc.transport.RpcChannelFutureListener operationComplete
WARNING: build channel:[id: 0x848e1435, L:null ! R:/192.168.180.67:8060] failed
Feb 04, 2026 6:43:15 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPoolObjectFactory wrap
SEVERE: failed to get result from stp
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.180.67:8060
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)

Feb 04, 2026 6:43:15 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPool getChannel
SEVERE: Unable to validate object
java.util.NoSuchElementException: Unable to validate object
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericOb^C

@Rocky

Rocky

04/13/2026, 1:21 PM

The logs you provided indicate that while your FE might now be starting (or attempting to), it is failing to communicate with the Backends (BE/CN nodes). Specifically, the error

Connection refused: /192.168.180.67:8060

means the FE is trying to reach a Backend at that IP, but the connection is being rejected. Since you moved volumes between VPCs, this is almost certainly an IP mismatch or a network isolation issue. 1. The Root Cause: Stale BE IPs in Metadata When you move a StarRocks cluster to a new VPC/environment, the Pod IP addresses usually change. StarRocks FEs store the IP addresses of the BEs in their metadata. * The FE is trying to talk to the "old" IP (

192.168.180.67

). * In your new VPC, the BEs likely have new IPs, or the network routing to the old subnet no longer exists. 2. How to fix this Step A: Check the actual BE status If you can now connect to the FE via MySQL (port 9030), run:

Copy code

sql
SHOW BACKENDS\G
-- Or if using shared-data (lake) mode:
SHOW COMPUTE NODES\G

* Check the

IP

and

Alive

columns. You will likely see

192.168.180.67

listed as

Alive: false

. Step B: Update BE/CN Registrations In Kubernetes, you should ideally be using FQDNs (Service names) instead of IPs, but StarRocks often resolves these to IPs in the metadata. If the IPs have changed, you must drop the "old" BEs and add the "new" ones: 1. Drop the ghost nodes:

Copy code

sql
    -- Replace with the IP/Port from your SHOW BACKENDS output
    ALTER SYSTEM DROP BACKEND "192.168.180.67:9050";

2. Add the nodes with their new identifiers: If you are using the StarRocks Operator, it should handle re-registration automatically if the heartbeats can sync. If it doesn't, you may need to manually add them using their new Pod IPs or Service DNS. Step C: Verify Network and Security Groups Since you are in a new VPC, ensure the following: * Security Groups: Port

(bRPC),

(BE Port), and

(Edit Log/Heartbeat) must be open between the FE subnet and the BE subnet. * Connection Test: From inside the FE pod, try to ping or telnet the BE:

Copy code

bash
    # Try to reach the BE's bRPC port
    timeout 2 bash -c '</dev/tcp/192.168.180.67/8060' && echo "port open" || echo "closed"

3. If the FE is still crashing (Exit 255) If those logs were from a brief moment before another crash, refer back to the Permission Fix: The

ls -al

you showed earlier proved the metadata was owned by

root

Copy code

bash
# Must be run inside the pod as root
chown -R starrocks:starrocks /opt/starrocks/fe/meta

Without this, the FE will eventually hit a write-operation and crash with Code 255 again. Summary Checklist for VPC Migration: 1. Permissions: Ensure

/opt/starrocks/fe/meta

is owned by the

starrocks

user (UID 1000). 2. Priority Networks: In

fe.conf

, update

priority_networks

to match your new VPC CIDR (e.g.,

priority_networks = 10.0.0.0/16

). 3. Clean up Backends: Use

SHOW BACKENDS

to identify and remove any nodes still pointing to the old VPC's IP range. References • administration/Meta_recovery.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • faq/Deploy_faq.md

Abhishek Yadav

04/13/2026, 1:22 PM

@Rocky not able to connect using mysql -h cmd

Rocky

04/13/2026, 1:23 PM

If you are unable to connect via

mysql -h

, it means either the FE process hasn't fully opened its query port (9030), or there is a network barrier in your new VPC. Here is a step-by-step troubleshooting guide: 1. Check if the Port is actually Listening Inside the FE pod, run:

Copy code

bash
netstat -tuln | grep 9030

* If nothing shows up: The FE hasn't finished starting or has crashed again. Check

fe.log

for the string

"thrift server started"

. * If it is listening: The problem is likely your VPC Network/Security Groups or the IP you are connecting to. 2. Check
priority_networks
(Critical for VPC moves) In your previous

fe.conf

snippet, I didn't see

priority_networks

. In a new VPC, the FE might be trying to bind to a dead IP address or the wrong network interface. Add this to your `fe.conf`:

Copy code

properties
# Match the CIDR of your new VPC subnets
priority_networks = 10.0.0.0/16

(Replace
10.0.0.0/16
with the actual CIDR of your new Kubernetes Pod network). 3. Verify the FE is the LEADER If the FE is stuck as a

FOLLOWER

(as your

ROLE

file indicated) and it cannot find a Leader, it will not open the MySQL port for business. Since you are recovering a single node, you must ensure the

bdbje_reset_election_group

fix was successful. Search your

fe.log

for:

Copy code

text
grep "transfer to LEADER" /opt/starrocks/fe/log/fe.log

* If found: The FE is ready. * If not found: The FE is still waiting for a leader. Re-read the logs for BDBJE errors. 4. Test Connectivity Locally While inside the FE pod, try to connect to itself using the loopback address:

Copy code

bash
mysql -h 127.0.0.1 -P 9030 -u root

* If this works: The FE is healthy. The issue is your VPC Security Group (ensure port 9030 is open) or your Kubernetes Service (ensure the LoadBalancer/NodePort is updated for the new VPC). * If this fails: The FE process is not fully initialized. 5. Check the "Catalog Ready" status Your previous logs showed the FE was trying to reach

192.168.180.67

(an old BE IP). If the FE is stuck waiting for metadata heartbeats from old backends, it might delay the startup of the MySQL service. What to do next: 1. Check the end of

/opt/starrocks/fe/log/fe.log

. Look for any lines containing

Exception

Error

. 2. If you see

Master not found

Wait for leader

, it means

bdbje_reset_election_group = true

didn't work or wasn't picked up. 3. Confirm you are using the correct New IP or Service DNS to connect. Quick Command to check status:

Copy code

bash
# See if the java process is still alive
ps aux | grep StarRocksFE

References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 1:25 PM

Copy code

root@kube-starrocks-fe-0:/opt/starrocks# mysql -h 127.0.0.1 -P 9030 -u root
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:9030' (111)
root@kube-starrocks-fe-0:/opt/starrocks# root@kube-starrocks-fe-0:/opt/starrocks# ps aux | grep StarRocksFE
root         517  0.0  0.0   3476  2004 pts/0    S+   21:25   0:00 grep --color=auto StarRocksFE
root@kube-starrocks-fe-0:/opt/starrocks#

@Rocky

Rocky

04/13/2026, 1:25 PM

The

ps aux

output confirms that the StarRocks FE process is not running. This is why your MySQL connection is being refused. The process likely started and then crashed or exited immediately. To find the exact cause, you must look at the logs. Since you are in a troubleshooting state (using a

sleep

command in the pod), follow these steps exactly: 1. Check the Log Files Run these commands to see the most recent errors:

Copy code

bash
# Check the standard output/error (this often has the 'Exit 255' reason)
tail -n 100 /opt/starrocks/fe/log/fe.out

# Check the main application log
tail -n 100 /opt/starrocks/fe/log/fe.log

# Check for warning/error logs
tail -n 100 /opt/starrocks/fe/log/fe.warn.log

2. Common Causes for the Process Dying Look for these specific error signatures in the logs: * java.io.IOException: Permission denied
: If you see this, your

chown

command from the previous step might not have covered everything. Run this again as root:

chown -R starrocks:starrocks /opt/starrocks/fe

* RepRefreshException
or
Master not found
: Because your

ROLE

FOLLOWER

, the FE is looking for a Leader. Since this is a single node in a new VPC, it will wait forever and then exit. * Fix: Ensure

bdbje_reset_election_group = true

is in your

fe.conf

. * FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: /192.x.x.x
: If the IP address the FE is picking up doesn't match the FQDN/IP in your metadata, it will crash. * Fix: Add

priority_networks = <your_new_pod_cidr>

(e.g.,

10.42.0.0/16

) to

fe.conf

. * Java Heap Space / OutOfMemoryError
: The FE might not have enough RAM assigned in

fe.conf

. 3. How to Start and Debug in Real-time Instead of using the

--daemon

flag, try starting it in the foreground to see the errors immediately on your terminal:

Copy code

bash
# Switch to the starrocks user first
su starrocks

# Start FE in foreground (remove --daemon)
/opt/starrocks/fe/bin/start_fe.sh

If the process crashes, the error will be printed directly to your screen. 4. Checklist for the current state: 1. Is
fe.conf
updated? Does it contain

bdbje_reset_election_group = true

? 2. Are permissions correct? Did you run

chown -R starrocks:starrocks /opt/starrocks/fe/meta

? 3. Is there a port conflict? Check if something else is using port 9010, 9020, or 9030 (though unlikely in a fresh pod). Please provide the last 20 lines of
/opt/starrocks/fe/log/fe.out
if it still fails to start. References • administration/Meta_recovery.md • administration/management/logs.md • faq/operation_maintenance_faq.md .** References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 1:27 PM

Copy code

@Rocky 2026-04-13 21:09:28.847+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local:9010]
2026-04-13 21:09:28.854+08:00 ERROR (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():498] Unmatched run mode between config file and version file: shared_nothing vs shared_data. will exit! 
2026-04-13 21:19:46.514+08:00 INFO (main|1) [StarRocksFE.start():137] StarRocks FE starting, version: 3.5.15-5abb1cb
2026-04-13 21:19:46.521+08:00 INFO (main|1) [NetUtils.getHosts():79] ipv6 link local address fe80:0:0:0:e2:7fff:fecd:4a91%eth0 is skipped
2026-04-13 21:19:46.523+08:00 INFO (main|1) [FrontendOptions.initAddrUseIp():269] check ip address: /192.168.124.73
2026-04-13 21:19:46.523+08:00 INFO (main|1) [FrontendOptions.initAddrUseIp():292] Use IP init local addr, IP: /192.168.124.73
2026-04-13 21:19:46.583+08:00 INFO (main|1) [ConsistencyChecker.initWorkTime():133] parsed startDate: 1970-01-01T23:00:00.000+0800, endDate: 1970-01-01T04:00:00.000+0800
2026-04-13 21:19:46.820+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role root[-1]
2026-04-13 21:19:46.824+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role db_admin[-2]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role cluster_admin[-3]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role user_admin[-4]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role security_admin[-106]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role public[-5]
2026-04-13 21:19:47.240+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [192.168.124.73:9010]
2026-04-13 21:19:47.248+08:00 ERROR (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():498] Unmatched run mode between config file and version file: shared_nothing vs shared_data. will exit! 
root@kube-starrocks-fe-0:/opt/starrocks#

@Rocky

Rocky

04/13/2026, 1:27 PM

The error message

Unmatched run mode between config file and version file: shared_nothing vs shared_data

is the reason your FE is exiting. The Problem Your existing metadata (stored in the volume you moved) was created as a Shared Data (Lake) cluster. However, the StarRocks FE defaults to Shared Nothing mode unless explicitly told otherwise. StarRocks will not allow a "Shared Nothing" process to start using "Shared Data" metadata. The Fix You must update your

fe.conf

to enable the correct mode. 1. Edit your
fe.conf
(usually located at

/opt/starrocks/fe/conf/fe.conf

): Add this line:

Copy code

properties
    run_mode = shared_data

2. Check Cloud Storage Settings: Since you are in a new VPC, your FE needs to be able to talk to your object storage (S3, MinIO, etc.). Ensure your storage credentials and endpoints are still valid in this new network. If you used

fe.conf

to define your storage, verify these:

Copy code

properties
    cloud_native_storage_type = S3  # or AZBLOB/HDFS
    # Ensure S3 endpoints/keys are updated if they changed for the new VPC

3. Permissions Check (Again): After editing the file, make sure the

starrocks

user still owns everything:

Copy code

bash
    chown -R starrocks:starrocks /opt/starrocks/fe

4. Restart the FE: Now try starting the FE again. Why this happened during VPC migration: When you moved the storage volumes, you brought the "Shared Data" identity of the old cluster with you. The new FE pods you spun up were likely using a default configuration (Shared Nothing). Note on Backends: In

shared_data

mode, StarRocks typically uses Compute Nodes (CN) instead of standard Backends (BE). * If you are using the StarRocks Operator, ensure your

StarRocksCluster

YAML has

runMode: shared_data

set. * The connection errors you saw earlier (

Connection refused: /192.168.180.67:8060

) were the FE trying to find its compute nodes at their old IPs. Once the FE starts successfully in

shared_data

mode, you will need to update the compute node registrations. Try adding
run_mode = shared_data
and let me know the result of the next
fe.log
output! References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md

Abhishek Yadav

04/13/2026, 1:36 PM

Copy code

root@kube-starrocks-fe-0:/opt/starrocks# su starrocks
This account is currently not available.
root@kube-starrocks-fe-0:/opt/starrocks# chown -R starrocks:starrocks /opt/starrocks/fe/meta
root@kube-starrocks-fe-0:/opt/starrocks# tail -n 20 /opt/starrocks/fe/log/fe.log
2026-04-13 21:09:28.435+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role db_admin[-2]
2026-04-13 21:09:28.435+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role cluster_admin[-3]
2026-04-13 21:09:28.435+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role user_admin[-4]
2026-04-13 21:09:28.436+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role security_admin[-106]
2026-04-13 21:09:28.436+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role public[-5]
2026-04-13 21:09:28.847+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local:9010]
2026-04-13 21:09:28.854+08:00 ERROR (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():498] Unmatched run mode between config file and version file: shared_nothing vs shared_data. will exit! 
2026-04-13 21:19:46.514+08:00 INFO (main|1) [StarRocksFE.start():137] StarRocks FE starting, version: 3.5.15-5abb1cb
2026-04-13 21:19:46.521+08:00 INFO (main|1) [NetUtils.getHosts():79] ipv6 link local address fe80:0:0:0:e2:7fff:fecd:4a91%eth0 is skipped
2026-04-13 21:19:46.523+08:00 INFO (main|1) [FrontendOptions.initAddrUseIp():269] check ip address: /192.168.124.73
2026-04-13 21:19:46.523+08:00 INFO (main|1) [FrontendOptions.initAddrUseIp():292] Use IP init local addr, IP: /192.168.124.73
2026-04-13 21:19:46.583+08:00 INFO (main|1) [ConsistencyChecker.initWorkTime():133] parsed startDate: 1970-01-01T23:00:00.000+0800, endDate: 1970-01-01T04:00:00.000+0800
2026-04-13 21:19:46.820+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role root[-1]
2026-04-13 21:19:46.824+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role db_admin[-2]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role cluster_admin[-3]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role user_admin[-4]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role security_admin[-106]
2026-04-13 21:19:46.825+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role public[-5]
2026-04-13 21:19:47.240+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [192.168.124.73:9010]
2026-04-13 21:19:47.248+08:00 ERROR (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():498] Unmatched run mode between config file and version file: shared_nothing vs shared_data. will exit! 
root@kube-starrocks-fe-0:/opt/starrocks#

@Rocky

Rocky

04/13/2026, 1:36 PM

Your FE is failing to start because it thinks it's a Shared-Nothing cluster (the default), but your existing metadata volume belongs to a Shared-Data (lake) cluster. Follow these steps to fix the configuration and handle the user permission issue: 1. Fix the
su starrocks
error The

starrocks

user in the container is likely configured with

/sbin/nologin

. To switch to that user for running the FE, use the

-s

flag to specify a shell:

Copy code

bash
su -s /bin/bash starrocks

2. Update
fe.conf
to
shared_data
mode Since the FE process is exiting with

shared_nothing vs shared_data

, you must explicitly set the run mode. Run this command as root to append the setting to your config file:

Copy code

bash
echo "run_mode = shared_data" >> /opt/starrocks/fe/conf/fe.conf

3. Verify Cloud Storage Config In

shared_data

mode, the FE must be able to talk to your object storage (S3/MinIO/Azure). Ensure your

fe.conf

also has the correct storage settings for your new VPC. For example:

Copy code

properties
cloud_native_storage_type = S3
# Ensure your S3/MinIO endpoints and keys are correct for the new network

4. Set Permissions and Start Ensure the

starrocks

user owns the directories, then start the FE:

Copy code

bash
# As root:
chown -R starrocks:starrocks /opt/starrocks/fe

# Switch to starrocks user and start:
su -s /bin/bash starrocks
/opt/starrocks/fe/bin/start_fe.sh --daemon

5. Check if it's Running Wait about 30 seconds and check the process and logs:

Copy code

bash
ps aux | grep StarRocksFE
tail -f /opt/starrocks/fe/log/fe.log

Once you see
thrift server started
or
StarRocksFE is now LEADER
in the logs, the MySQL port (9030) will open, and you can connect:

Copy code

bash
mysql -h 127.0.0.1 -P 9030 -u root

Important Note for Kubernetes: If you are using the StarRocks Operator, you should update your

StarRocksCluster

YAML file to include

runMode: shared_data

. Otherwise, if the pod restarts, your manual changes to

fe.conf

inside the pod will be lost. References • administration/management/FE_configuration.md • quick_start/shared-data.md • deployment/shared_data/minio.md References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md

Abhishek Yadav

04/13/2026, 1:48 PM

Copy code

tail -f /opt/starrocks/fe/log/fe.log

oot@kube-starrocks-fe-0:/opt/starrocks# su -s /bin/bash starrocks
starrocks@kube-starrocks-fe-0:/opt/starrocks$ echo "run_mode = shared_data" >> /opt/starrocks/fe/conf/fe.conf
starrocks@kube-starrocks-fe-0:/opt/starrocks$ 
starrocks@kube-starrocks-fe-0:/opt/starrocks$ chown -R starrocks:starrocks /opt/starrocks/fe
starrocks@kube-starrocks-fe-0:/opt/starrocks$ su -s /bin/bash starrocks
/opt/starrocks/fe/bin/start_fe.sh --daemon
Password: 
su: Authentication failure
starrocks@kube-starrocks-fe-0:/opt/starrocks$ sudo su /opt/starrocks/fe/bin/start_fe.sh --daemon
bash: sudo: command not found
starrocks@kube-starrocks-fe-0:/opt/starrocks$ sudo su
bash: sudo: command not found
starrocks@kube-starrocks-fe-0:/opt/starrocks$ exit
exit
root@kube-starrocks-fe-0:/opt/starrocks# /opt/starrocks/fe/bin/start_fe.sh --daemon
Frontend running as process 487. Stop it first.
root@kube-starrocks-fe-0:/opt/starrocks# ps aux | grep StarRocksFE
starroc+     487 18.4  0.5 13748672 387208 pts/0 Sl   21:46   0:08 /lib/jvm/java-17-openjdk/bin/java -Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:/opt/starrocks/fe/log/fe.gc.log.20260413-214649:time -XX:ErrorFile=/opt/starrocks/fe/log/hs_err_pid%p.log -Djava.security.policy=/opt/starrocks/fe/conf/udf_security.policy com.starrocks.StarRocksFE
root        1212  0.0  0.0   3476  1804 pts/0    S+   21:47   0:00 grep --color=auto StarRocksFE
root@kube-starrocks-fe-0:/opt/starrocks# tail -f /opt/starrocks/fe/log/fe.log
2026-04-13 21:46:53.193+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role db_admin[-2]
2026-04-13 21:46:53.194+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role cluster_admin[-3]
2026-04-13 21:46:53.194+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role user_admin[-4]
2026-04-13 21:46:53.194+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role security_admin[-106]
2026-04-13 21:46:53.195+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role public[-5]
2026-04-13 21:46:53.772+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [192.168.124.75:9010]
2026-04-13 21:46:53.782+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():508] Current run_mode is shared_data
2026-04-13 21:46:53.782+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():513] Got role: FOLLOWER, node name: kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1776075864420 and run_mode: shared_data
2026-04-13 21:46:53.786+08:00 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():340] skip check local environment because helper node and local node are identical.
2026-04-13 21:46:53.819+08:00 INFO (main|1) [BDBEnvironment.setupEnvironment():270] start to setup bdb environment for 1 times

root@kube-starrocks-fe-0:/opt/starrocks# mysql -h 127.0.0.1 -P 9030 -u root
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:9030' (111)
root@kube-starrocks-fe-0:/opt/starrocks#

@Rocky

Rocky

04/13/2026, 1:49 PM

You are very close! The FE process is running, but it is stuck in a "FOLLOWER" state, waiting for a Leader FE that no longer exists (due to the VPC/IP change). In this state, the FE will not open the MySQL port (9030) until it finds a leader or becomes one. To fix this, you need to force this FE to become the LEADER by resetting the election group. 1. Stop the current FE process Since you are logged in as

root

, kill the existing process first:

Copy code

bash
kill -9 487

2. Add the Recovery Flag to
fe.conf
Add the

bdbje_reset_election_group

parameter. This tells the FE to ignore its old cluster metadata and start as the new Leader. Run this as `root`:

Copy code

bash
echo "bdbje_reset_election_group = true" >> /opt/starrocks/fe/conf/fe.conf

3. Start the FE again Start it as the

starrocks

user:

Copy code

bash
# Switch to starrocks user
su -s /bin/bash starrocks

# Start FE
/opt/starrocks/fe/bin/start_fe.sh --daemon

4. Verify and Connect Check the

fe.log

again. You are looking for a message like:

[TransferService.run():...] transfer from UNKNOWN to LEADER

Once you see it has become LEADER, you can connect via MySQL:

Copy code

bash
mysql -h 127.0.0.1 -P 9030 -u root

5. Important Post-Recovery Step Once the MySQL connection works and the cluster is up: 1. Remove the line

bdbje_reset_election_group = true

from your

/opt/starrocks/fe/conf/fe.conf

file. If you leave it there, the FE will reset itself every time it restarts, which can cause metadata issues later. 2. If you have other FE nodes to add, you will now add them as fresh nodes using this FE as the

--helper

. 3. Check your Compute Nodes (CN): Since this is

shared_data

mode, you will likely need to re-register your compute nodes using their new IPs:

Copy code

sql
    -- Run this in MySQL once connected
    SHOW COMPUTE NODES;
    -- If they are 'Dead' or have old IPs, drop and add them:
    ALTER SYSTEM DROP COMPUTE NODE "old_ip:9050";
    ALTER SYSTEM ADD COMPUTE NODE "new_ip:9050";

Try the
bdbje_reset_election_group
fix and let me know if port 9030 opens up! References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/shared_data/minio.md References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 1:50 PM

Copy code

root@kube-starrocks-fe-0:/opt/starrocks# mysql -h 127.0.0.1 -P 9030 -u root
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:9030' (111)
root@kube-starrocks-fe-0:/opt/starrocks# kill -9 487
root@kube-starrocks-fe-0:/opt/starrocks# echo "bdbje_reset_election_group = true" >> /opt/starrocks/fe/conf/fe.conf
root@kube-starrocks-fe-0:/opt/starrocks# su -s /bin/bash starrocks
starrocks@kube-starrocks-fe-0:/opt/starrocks$ /opt/starrocks/fe/bin/start_fe.sh --daemon
starrocks@kube-starrocks-fe-0:/opt/starrocks$ mysql -h 127.0.0.1 -P 9030 -u root
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:9030' (111)
starrocks@kube-starrocks-fe-0:/opt/starrocks$ tail -f /opt/starrocks/fe/log/fe.log
        at com.starrocks.journal.bdbje.BDBJEJournal.read(BDBJEJournal.java:137) ~[starrocks-fe.jar:?]
        at com.starrocks.staros.StarOSBDBJEJournalSystem.replayTo(StarOSBDBJEJournalSystem.java:191) ~[starrocks-fe.jar:?]
        at com.starrocks.staros.StarOSBDBJEJournalSystem.onBecomeLeader(StarOSBDBJEJournalSystem.java:110) ~[starrocks-fe.jar:?]
        at com.staros.journal.DelegateJournalSystem.onBecomeLeader(DelegateJournalSystem.java:79) ~[starmanager-3.5.3.jar:?]
        at com.staros.manager.StarManager.becomeLeader(StarManager.java:188) ~[starmanager-3.5.3.jar:?]
        at com.starrocks.staros.StarMgrServer.becomeLeader(StarMgrServer.java:215) ~[starrocks-fe.jar:?]
        at com.starrocks.staros.StarMgrServer$1.transferToLeader(StarMgrServer.java:100) ~[starrocks-fe.jar:?]
        at com.starrocks.ha.StateChangeExecutor.runOneCycle(StateChangeExecutor.java:103) ~[starrocks-fe.jar:?]
        at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?]
2026-04-13 21:49:55.346+08:00 WARN (stateChangeExecutor|110) [StarOSBDBJEJournalSystem.onBecomeLeader():118] star mgr prepare journal failed before becoming leader, Can not find the key[1] in [49707063]: key too small.

@Rocky

Rocky

04/13/2026, 1:51 PM

The error

star mgr prepare journal failed before becoming leader, Can not find the key[1] in [49707063]: key too small

indicates that while the main FE metadata group was reset, the StarManager (StarOS) component—which handles shard and tablet mapping in shared-data mode—is failing to start. It is trying to replay its journal from ID

, but your BDBJE database only contains entries starting from

49707063

. This usually happens if the StarManager image file is missing or inaccessible, forcing it to try replaying from the very beginning of time. Please follow these steps to resolve this: 1. Check for the StarManager Image StarManager needs its own checkpoint (image) to know where to start replaying the journal. Check if this directory exists and contains an image file:

Copy code

bash
ls -lh /opt/starrocks/fe/meta/image/starmgr/

* If it is empty: You are likely missing a critical metadata file from your migration. You should look for the

starmgr

image in your old volume/backup. * If it has files: Ensure the

starrocks

user has full permissions to it:

Copy code

bash
    chown -R starrocks:starrocks /opt/starrocks/fe/meta/image/starmgr/

2. Force StarManager to Reset If the main FE reset didn't propagate to the StarManager, you may need to explicitly reset the StarManager's election group as well. In StarRocks 3.x, use this parameter in `fe.conf`: 1. Stop the FE. 2. Add this to
fe.conf
:

Copy code

properties
    # This resets the StarManager component specifically
    starmgr_bdbje_reset_election_group = true

_(Note: If

starmgr_bdbje_reset_election_group

is not recognized, try

bdbje_reset_election_group = true

again but ensure the helper node is the local node)._ 3. Ensure the FE starts with itself as helper: Make sure your

fe.conf

does not have a

meta_helper

pointing to an old IP. When recovering the first node, it should only see itself. 3. Clear StarManager BDB (Last Resort) If you have a valid StarManager image but the BDB journal is hopelessly corrupted or "key too small" persists, you can try clearing the StarManager's BDB logs so it forces a reload from the image: Warning: Only do this if you have an image file in
meta/image/starmgr/
.

Copy code

bash
# 1. Stop FE
# 2. Back up the directory first!
cp -r /opt/starrocks/fe/meta/starmgr /opt/starrocks/fe/meta/starmgr_backup
# 3. Clear the BDB logs for StarManager
rm -rf /opt/starrocks/fe/meta/starmgr/bdb/*
# 4. Start FE

4. Verify Local IP and Helper In your logs, I see:

get helper nodes: [192.168.124.75:9010]

But your local IP is

192.168.124.73

. If you are trying to recover a single-node cluster, the FE should not be looking at

.75

. 1. Check your

fe.conf

for

priority_networks

. Ensure it matches the new

192.168.124.0/24

subnet. 2. Start the FE manually one time with the

--helper

flag pointing to itself to force the metadata to bind to the new IP:

Copy code

bash
    /opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon

Summary of what to try now: 1. Add

starmgr_bdbje_reset_election_group = true

fe.conf

. 2. Ensure

bdbje_reset_election_group = true

is still there. 3. Start the FE using:

/opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon

4. Tail the log and look for

transfer from UNKNOWN to LEADER

. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • deployment/shared_data/gcs.md • deployment/shared_data/s3.md

Abhishek Yadav

04/13/2026, 1:58 PM

@Rocky any cmd to forcefull make pod a leader incase we have only 1 fe pod ?

Rocky

04/13/2026, 1:59 PM

To forcefully make a single FE pod the Leader when it is stuck as a FOLLOWER or failing during the transition (especially in

shared_data

mode), you must reset the election groups for both the FE and the StarManager (StarOS) components. The error

Can not find the key[1]... key too small

happens because the StarManager is trying to replay its journal from the beginning (ID 1) but can't find the logs. This usually means its checkpoint (image) is missing or out of sync. Follow these exact steps to force the recovery: 1. Stop the FE Kill the process if it's running:

Copy code

bash
# As root
ps aux | grep StarRocksFE
kill -9 <PID>

2. Configure Force-Leader Flags Add both of these flags to your

/opt/starrocks/fe/conf/fe.conf

. The second one is specifically for the StarManager component that is currently failing in your logs:

Copy code

properties
# Force the main FE to become Leader
bdbje_reset_election_group = true

# Force the StarManager (StarOS) to become Leader
starmgr_bdbje_reset_election_group = true

3. Clear StarManager BDB Logs (If needed) If the error

key too small

persists after Step 2, it means the BDB logs for the StarManager are inconsistent with its image. You can force it to reload from its last image (checkpoint) by clearing its BDB directory: Caution: Ensure you have files in

/opt/starrocks/fe/meta/image/starmgr/

before doing this.

Copy code

bash
# Back up and clear the StarManager BDB logs
cp -r /opt/starrocks/fe/meta/starmgr/bdb /opt/starrocks/fe/meta/starmgr/bdb_backup
rm -rf /opt/starrocks/fe/meta/starmgr/bdb/*

4. Start the FE as the
starrocks
user Use the

--helper

flag pointing to itself to ensure it doesn't try to look for old IPs:

Copy code

bash
su -s /bin/bash starrocks
/opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon

5. Verify and Cleanup Check the log:

Copy code

bash
tail -f /opt/starrocks/fe/log/fe.log

Look for:

[TransferService.run():...] transfer from UNKNOWN to LEADER

. Once the FE is running and you can connect via MySQL (

mysql -P 9030

), you must remove the two
reset_election_group
lines from

fe.conf

. If you don't, the FE will reset its metadata every time it restarts, which will cause data loss or cluster instability in the future. Summary Checklist for Single Pod: 1. run_mode = shared_data
must be in

fe.conf

. 2. bdbje_reset_election_group = true
forces FE Leader. 3. starmgr_bdbje_reset_election_group = true
forces StarManager Leader. 4. --helper 127.0.0.1:9010
prevents it from searching for old peers. References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 2:00 PM

Copy code

@kube-starrocks-fe-0:/opt/starrocks$ ls /opt/starrocks/fe/meta/image/starmgr/
starrocks@kube-starrocks-fe-0:/opt/starrocks$ /opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon
starrocks@kube-starrocks-fe-0:/opt/starrocks$ tail -f /opt/starrocks/fe/log/fe.log
2026-04-13 21:59:58.591+08:00 INFO (main|1) [ConsistencyChecker.initWorkTime():133] parsed startDate: 1970-01-01T23:00:00.000+0800, endDate: 1970-01-01T04:00:00.000+0800
2026-04-13 21:59:58.827+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role root[-1]
2026-04-13 21:59:58.832+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role db_admin[-2]
2026-04-13 21:59:58.832+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role cluster_admin[-3]
2026-04-13 21:59:58.833+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role user_admin[-4]
2026-04-13 21:59:58.833+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role security_admin[-106]
2026-04-13 21:59:58.833+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role public[-5]
2026-04-13 21:59:59.244+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [127.0.0.1:9010]
2026-04-13 21:59:59.251+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():293] find ROLE and VERSION file in local, ignore helper nodes: [127.0.0.1:9010]
2026-04-13 21:59:59.252+08:00 ERROR (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():498] Unmatched run mode between config file and version file: shared_nothing vs shared_data. will exit!

@Rocky

Rocky

04/13/2026, 2:01 PM

The error

Unmatched run mode between config file and version file: shared_nothing vs shared_data

indicates that your

fe.conf

is currently interpreted as

shared_nothing

(the default), but your existing metadata in the

meta/

directory was created in

shared_data

mode. This likely happened because your manual

echo

command into

fe.conf

was lost (likely due to a Kubernetes pod restart) or the file was overwritten. 1. Fix the Run Mode Mismatch You must ensure

run_mode = shared_data

is explicitly set in your

fe.conf

. Check the file content first:

Copy code

bash
grep "run_mode" /opt/starrocks/fe/conf/fe.conf

If it is missing or says

shared_nothing

, fix it:

Copy code

bash
# As root
echo "run_mode = shared_data" >> /opt/starrocks/fe/conf/fe.conf

(Note: If you are using Kubernetes, editing files inside the pod is temporary. You should ideally update your ConfigMap or Helm values, otherwise this will happen again every time the pod restarts.) 2. The StarManager Image Problem Your previous logs showed:

Can not find the key[1] in [49707063]: key too small

. And your

ls

command showed that

/opt/starrocks/fe/meta/image/starmgr/

is empty. This is a critical issue. In

shared_data

mode, the StarManager (StarOS) component stores the mapping of tablets to object storage. If the image (checkpoint) is missing and the BDB logs only start at ID

49707063

, the FE doesn't know what happened between journal ID

and

49707063

. To fix this, you have two options: Option A: Find the missing metadata (Recommended) Check if the metadata is in a different path or if you have a backup of the

meta/

directory. You are looking for a file named

image.<number>

inside a

starmgr

folder. Option B: Force Reinitialization (Data Loss Risk) If this is a fresh cluster or you don't mind losing the metadata (and recreating tables), you could clear the metadata. However, if you want to keep the data, you must try to force the StarManager to ignore the missing gap: 1. Stop the FE. 2. Add these recovery flags to

fe.conf

(ensure they are at the bottom):

Copy code

properties
    run_mode = shared_data
    bdbje_reset_election_group = true
    starmgr_bdbje_reset_election_group = true

3. Start the FE with the helper flag:

Copy code

bash
    su -s /bin/bash starrocks
    /opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon

3. Verify the Version file If you want to be 100% sure what the metadata thinks the mode is, check this file:

Copy code

bash
cat /opt/starrocks/fe/meta/image/VERSION | grep runMode

It should say

runMode=shared_data

. If your

fe.conf

matches this, the error will go away. Summary Checklist: 1. Check
fe.conf
: Ensure

run_mode = shared_data

is there. 2. Check
VERSION
: Ensure it says

runMode=shared_data

. 3. Check
starmgr
image: If the directory is truly empty, the FE will struggle to start because it has lost the "starting point" for its StarManager journals. You may need to restore that directory from your old PV/storage. Once you fix the
run_mode
in the config file, the "Unmatched run mode" error will disappear, and we can see if the StarManager can finally become Leader. References • administration/management/FE_configuration.md • administration/Meta_recovery.md • deployment/shared_data/minio.md References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md

Rocky

04/13/2026, 2:01 PM

• deployment/shared_data/minio.md

Abhishek Yadav

04/13/2026, 2:04 PM

Copy code

starrocks@kube-starrocks-fe-0:/opt/starrocks$ grep "run_mode" /opt/starrocks/fe/conf/fe.conf
starrocks@kube-starrocks-fe-0:/opt/starrocks$ exit
exit
root@kube-starrocks-fe-0:/opt/starrocks# echo "run_mode = shared_data" >> /opt/starrocks/fe/conf/fe.conf
root@kube-starrocks-fe-0:/opt/starrocks# /opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon
root@kube-starrocks-fe-0:/opt/starrocks# cat /opt/starrocks/fe/meta/image/VERSION | grep runMode
runMode=shared_data
root@kube-starrocks-fe-0:/opt/starrocks#

@Rocky

Rocky

04/13/2026, 2:05 PM

The error

Can not find the key[1] in [49707063]: key too small

confirms that the StarManager (StarOS) component is trying to start from scratch (Journal ID 1) because it cannot find its checkpoint (image) file, but the underlying BDB database already has data starting from ID

49707063

. In

shared_data

mode, StarRocks has two metadata stores. You fixed the main FE "run mode" mismatch, but the StarManager metadata is still broken or missing. 1. Find the Missing StarManager Image The path you checked earlier (

/opt/starrocks/fe/meta/image/starmgr/

) was likely the wrong directory. In StarRocks 3.x, the StarManager metadata is usually stored here:

Copy code

bash
ls -R /opt/starrocks/fe/meta/starmgr

Specifically, look for a file starting with

image.

inside

/opt/starrocks/fe/meta/starmgr/image/

. * If the directory is empty: You have lost the mapping of tablets to object storage (S3/GCS). You must restore this directory from a backup or your previous persistent volume. * If files exist: It is likely a permissions issue. Run:

Copy code

bash
    chown -R starrocks:starrocks /opt/starrocks/fe/meta/starmgr

2. Force the StarManager to become Leader Since you have already added

run_mode = shared_data

, you now need to ensure the following two lines are in your

fe.conf

to force the election:

Copy code

properties
bdbje_reset_election_group = true
starmgr_bdbje_reset_election_group = true

3. Critical Fix: Clear StarManager Journals (If Image exists) If you found an image file in Step 1 but still get the

key too small

error, it means the StarManager's BDB journals are "ahead" of what the system expects or are corrupted. You can force it to reload from the image by clearing the StarManager's journal logs:

Copy code

bash
# 1. Stop the FE process
# 2. Delete ONLY the StarManager journals (NOT the main FE journals)
rm -rf /opt/starrocks/fe/meta/starmgr/bdb/*

# 3. Start FE again
/opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon

4. If this is a "Start Fresh" scenario If you do not care about the data currently in the cluster and just want to get the FE running: 1. Stop FE. 2.

rm -rf /opt/starrocks/fe/meta/*

3. Ensure

run_mode = shared_data

and your S3/MinIO configs are in

fe.conf

. 4. Start FE. It will initialize a brand new empty metadata set. 5. Check the logs again After running the start command, watch the logs. If it still fails, please share the new error. We are looking for:

[StarOSBDBJEJournalSystem.onBecomeLeader():...] transfer to LEADER

Warning: If you are in a Kubernetes environment, remember that changes made with

echo >> fe.conf

inside the pod will disappear if the pod restarts. You should update your

ConfigMap

values.yaml

once you confirm the fix. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md • deployment/shared_data/hdfs.md • quick_start/shared-data.md • administration/management/BE_configuration.md • deployment/shared_data/azure.md • deployment/shared_data/gcs.md • deployment/helm.md • administration/Management.md • deployment/shared_data/shared_data.md

Rocky

04/13/2026, 2:05 PM

Rocky

04/13/2026, 2:05 PM

Rocky

04/13/2026, 2:05 PM

Rocky

04/13/2026, 2:05 PM

Abhishek Yadav

04/13/2026, 2:13 PM

Copy code

root@kube-starrocks-fe-0:/opt/starrocks/fe/meta# ls
bdb  image  lost+found
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta# cd image/
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# ls
ROLE  VERSION  starmgr  v2
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# cat ROLE 
#Mon Apr 13 21:57:43 CST 2026
role=FOLLOWER
hostType=
name=192.168.124.76_9010_1776088663466
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# Cat Version
bash: Cat: command not found
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# cat VERSION
#Mon Apr 13 18:24:24 CST 2026
clusterId=1793171692
runMode=shared_data
token=xxxxxx
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# cd starmgr
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image/starmgr# ls
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image/starmgr# ls -la
total 8
drwxr-xr-x. 2 starrocks starrocks 4096 Apr 13 18:24 .
drwxr-xr-x. 4 starrocks starrocks 4096 Apr 13 18:24 ..
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image/starmgr# cd ..
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# ls
ROLE  VERSION  starmgr  v2
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# cd v2/
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image/v2# ls
root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image/v2#

How to make it Leader instead of follower @Rocky

Rocky

04/13/2026, 2:13 PM

To make your single FE pod become the Leader and resolve the StarManager "key too small" error, you must force both the main FE and the StarManager (StarOS) to reset their election groups. The error you are seeing (

key too small

) happens because the StarManager is trying to replay journals from the beginning (ID 1), but its image (checkpoint) is missing from the directory, so it looks at the BDB logs which start much later (at ID 49707063). Step 1: Fix the StarManager Image/BDB Conflict You checked

/opt/starrocks/fe/meta/image/starmgr/

, but in StarRocks 3.x, the StarManager metadata is usually one level up. Check this path:

Copy code

bash
ls -la /opt/starrocks/fe/meta/starmgr/image/

* If this directory is empty: This is why you get the "key too small" error. The StarManager has no starting point (image) but has existing logs. * Action: If you cannot find a backup of the image file, you must clear the StarManager's BDB logs to allow it to start a new "clean" state (Note: This may cause loss of tablet-to-S3 mappings if this was an existing production cluster).

Copy code

bash
    # Only if the image directory above is empty and you have no backup
    rm -rf /opt/starrocks/fe/meta/starmgr/bdb/*

Step 2: Configure Force-Leader Flags Add these lines to your

/opt/starrocks/fe/conf/fe.conf

. This tells both the main FE and the StarOS component to ignore previous election results and become the Leader immediately.

Copy code

properties
run_mode = shared_data
bdbje_reset_election_group = true
starmgr_bdbje_reset_election_group = true

Step 3: Start the FE Start the FE using itself as the helper to ensure it doesn't look for other nodes:

Copy code

bash
# Ensure you are the starrocks user
/opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon

Step 4: Verify and Cleanup 1. Check the log:

Copy code

bash
    tail -f /opt/starrocks/fe/log/fe.log

Look for:

[TransferService.run():...] transfer from UNKNOWN to LEADER

. 2. Verify via MySQL:

Copy code

sql
    -- Connect via mysql -P 9030
    SHOW FRONTENDS;

The

Role

column should now say

LEADER

. 3. IMPORTANT: Once the FE is running as Leader, remove the two
reset_election_group
lines from

fe.conf

. If you leave them, the FE will reset its metadata every time the pod restarts. Why was it stuck as FOLLOWER? Your

ROLE

file showed an old IP (

192.168.124.76

). In Kubernetes, if the pod IP changes, the FE thinks it is a new node and tries to find the "old" Leader at that old IP. Because you only have one pod, there is no leader to find, and it stays a FOLLOWER forever. The

bdbje_reset_election_group

flag is the only way to break this loop. References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 2:21 PM

@Rocky how to validate the last leader pod ?

Rocky

04/13/2026, 2:21 PM

To validate which pod was the "Last Leader" (the one with the most recent data), you can use three methods. Since you are in a recovery situation, the VLSN method is the most reliable. Method 1: The "Gold Standard" (VLSN Check) The node with the highest lastVLSN has the most up-to-date metadata. If a pod was the Leader, it will almost always have the highest VLSN. Run this command inside the pod (the JAR version may vary slightly based on your StarRocks version):

Copy code

bash
# 1. Find the BDB JAR file
ls /opt/starrocks/fe/lib/starrocks-bdb-je-*.jar

# 2. Run the DbPrintLog utility (replace the JAR name with yours)
java -jar /opt/starrocks/fe/lib/starrocks-bdb-je-18.3.16.jar DbPrintLog -h /opt/starrocks/fe/meta/bdb/ -vd | tail -n 20

What to look for: Look for the line:

file 0x... numRepRecords = ... firstVLSN = ... lastVLSN = 2,999,583

. Compare this

lastVLSN

number across all your FE pods/volumes. The pod with the largest number is your "Last Leader" (or the most up-to-date Follower). ────────── Method 2: Check the
ROLE
file Each FE records its last known state in the

ROLE

file. You already checked this on one pod and saw

role=FOLLOWER

Copy code

bash
cat /opt/starrocks/fe/meta/image/ROLE

* If it says

role=LEADER

, that pod was the Leader when it last shut down gracefully. * If all pods say

role=FOLLOWER

, it means the Leader likely crashed or the pods haven't been able to communicate to elect a new one. ────────── Method 3: Search the Logs You can search the historical logs for the last time any pod successfully became the leader.

Copy code

bash
grep "transfer to LEADER" /opt/starrocks/fe/log/fe.log

Or check for the StarManager (StarOS) leader transition:

Copy code

bash
grep "become leader" /opt/starrocks/fe/log/fe.log

Why this matters for you: In your previous message, your

ROLE

file showed:

name=192.168.124.76_9010_1776088663466

role=FOLLOWER

If you are running a single-pod cluster and it says

FOLLOWER

, it means the metadata is "locked" looking for a Leader that doesn't exist (likely an old pod with a different IP). This is why you must use the

bdbje_reset_election_group = true

flag to force this pod to take over as the new Leader. Recommendation: If you only have one pod/volume, that volume is your leader. Use the reset flags mentioned in the previous step to force it to start. References • administration/Meta_recovery.md • deployment/deploy_manually.md

Abhishek Yadav

04/13/2026, 2:40 PM

@Rocky the pod which were leader before I am getting

FE service kube-starrocks-fe-service.starrocks:9030 is not alive yet!

Rocky

04/13/2026, 2:40 PM

The message

FE service kube-starrocks-fe-service.starrocks:9030 is not alive yet!

is a health check error. It means the StarRocks process has started, but it is not yet ready to accept connections because it is stuck in the initialization phase. In a single-pod Kubernetes setup, this is almost always because the FE is waiting to become the Leader, but its internal metadata (BDBJE) thinks it is a Follower looking for an old IP address. 1. Check the true error in the logs Inside the pod, look for the actual Java stack trace or "Waiting" message:

Copy code

bash
tail -n 100 /opt/starrocks/fe/log/fe.log

You are looking for: *

wait globalStateMgr to be ready. FE type: UNKNOWN. is ready: false

* Or the StarManager error you saw earlier:

key too small

. 2. Force the "Last Leader" to take over Even if this pod was the leader before, its IP address likely changed when it was restarted in Kubernetes. BDBJE is very sensitive to IP changes. To fix this, you must force a "Metadata Reset" so the pod realizes it is the new Leader. Add these three lines to your `/opt/starrocks/fe/conf/fe.conf`:

Copy code

properties
# Force the main FE metadata to reset election
bdbje_reset_election_group = true

# Force the StarOS (shared-data) metadata to reset election
starmgr_bdbje_reset_election_group = true

# Ensure shared_data mode is active
run_mode = shared_data

3. Clear the StarManager "Key Too Small" error As seen in your previous logs, your StarManager (the component that manages S3/Shared-Data) is corrupted because its checkpoint (image) is missing but its logs (BDB) are at a very high ID. If the FE still won't start after Step 2, you must clear the StarManager's journal logs so it can start fresh (this is safe if you are restoring a single-node cluster):

Copy code

bash
# 1. Stop the FE (if it's running)
# 2. Clear ONLY the StarManager's BDB journals (NOT the main FE journals)
rm -rf /opt/starrocks/fe/meta/starmgr/bdb/*

# 3. Start the FE
/opt/starrocks/fe/bin/start_fe.sh --helper 127.0.0.1:9010 --daemon

4. Verify Success Once you run the start command: 1. Watch the log:

tail -f /opt/starrocks/fe/log/fe.log

. 2. Wait for the line:

[TransferService.run():...] transfer from UNKNOWN to LEADER

. 3. Check Port 9030: Run

netstat -tuln | grep 9030

. If it's open, the service is alive. 4. CRITICAL: After the FE is healthy and you can log in via MySQL, remove the

bdbje_reset_election_group

and

starmgr_bdbje_reset_election_group

lines from

fe.conf

. If you don't, the FE will reset itself every time the pod restarts. Why is this happening? In Kubernetes, when a pod restarts, it often gets a new IP. StarRocks stores the "Leader's IP" in its metadata. If the pod starts up and sees its IP doesn't match the one in the metadata, it assumes it is a Follower and waits forever for the "Old Leader" (the old IP) to come back online. Port 9030 will never open as long as the node is in a

FOLLOWER

UNKNOWN

state. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md

Abhishek Yadav

04/13/2026, 2:43 PM

Copy code

2026-04-13 22:42:43.001+08:00 WARN (main|1) [NodeMgr.getFeNodeTypeAndNameFromHelpers():558] failed to get fe node type from helper node: 127.0.0.1:9010.
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
        at sun.nio.ch.Net.connect(Net.java:591) ~[?:?]
        at sun.nio.ch.Net.connect(Net.java:580) ~[?:?]
        at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593) ~[?:?]
        at java.net.Socket.connect(Socket.java:633) ~[?:?]
        at java.net.Socket.connect(Socket.java:583) ~[?:?]
        at sun.net.NetworkClient.doConnect(NetworkClient.java:183) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:533) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:638) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:283) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:386) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:408) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1329) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1262) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1148) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1077) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1717) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1641) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
        at com.starrocks.server.NodeMgr.getFeNodeTypeAndNameFromHelpers(NodeMgr.java:531) ~[starrocks-fe.jar:?]
        at com.starrocks.server.NodeMgr.getClusterIdAndRoleOnStartup(NodeMgr.java:380) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1179) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:143) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:93) ~[starrocks-fe.jar:?]
2026-04-13 22:42:43.011+08:00 WARN (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():381] current node is not added to the group. please add it first. sleep 5 seconds and retry, current helper nodes: [127.0.0.1:9010]
root@kube-starrocks-fe-4:/opt/starrocks#

@Rocky

Rocky

04/13/2026, 2:43 PM

The

Connection refused

error to

127.0.0.1:9010

occurs because you are trying to use the FE pod as its own "helper" before the service has actually started. In a recovery situation where you want to force a pod to become the Leader, you should not use the

--helper

flag. Additionally, the error

current node is not added to the group

confirms that the IP address or Node Name of your current pod (

kube-starrocks-fe-4

) does not match what is stored in the metadata (which we saw earlier was

192.168.124.76

). To force this pod to become the Leader, follow these exact steps: 1. Modify
fe.conf
Edit

/opt/starrocks/fe/conf/fe.conf

and add these recovery flags. This tells StarRocks to ignore the old IP/group membership and start a new group with this pod as the Leader.

Copy code

properties
# Force the main metadata to ignore the old group and become Leader
bdbje_reset_election_group = true

# Since you are in shared_data mode, force the StarManager to reset as well
starmgr_bdbje_reset_election_group = true

*2. Start the FE without the helper flag* Do not point to

127.0.0.1

. Start the FE normally:

Copy code

bash
# Ensure no other FE process is running
ps ux | grep StarRocksFE | grep -v grep | awk '{print $2}' | xargs kill -9

# Start normally
/opt/starrocks/fe/bin/start_fe.sh --daemon

3. Monitor the startup log Watch the log carefully for the transition to Leader:

Copy code

bash
tail -f /opt/starrocks/fe/log/fe.log

What you want to see: *

[BDBEnvironment.initBDBEnvironment():...] reset bdbje election group

[TransferService.run():...] transfer from UNKNOWN to LEADER

[StarRocksFE.start():...] StarRocksFE started

4. Critical Cleanup Once the FE is running and you can connect via MySQL (

mysql -h 127.0.0.1 -P 9030 -u root

): 1. Remove the flags: Open

fe.conf

and delete the two lines you added (

bdbje_reset_election_group

and

starmgr_bdbje_reset_election_group

). 2. If you leave these flags in the config, the FE will wipe and reset its election group every time it restarts, which will prevent you from adding other followers later. Why did
--helper 127.0.0.1
fail? The

--helper

flag is used when a new node wants to join an existing cluster. Since your node is the only one and is currently "stuck," there is no active Leader at

127.0.0.1:9010

to talk to. By using

bdbje_reset_election_group

, you are telling the node: "Forget everyone else, you are the boss now." References • administration/Meta_recovery.md

Abhishek Yadav

04/13/2026, 2:50 PM

Copy code

root@kube-starrocks-fe-4:/opt/starrocks/fe/meta# tail -f  fe.warn.log.20260412-1
 2026-04-12 18:12:28.918+08:00 ERROR (statistics meta manager|28) [Daemon.run():100] daemon thread got exception. name: statistics meta manager
 java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:496)
        at com.starrocks.statistic.StatisticSQLBuilder.buildDropTableInvalidPartitionSQL(StatisticSQLBuilder.java:299)
        at com.starrocks.statistic.StatisticExecutor.dropTableInvalidPartitionStatistics(StatisticExecutor.java:300)
        at com.starrocks.statistic.AnalyzeMgr.clearStalePartitionStats(AnalyzeMgr.java:639)
        at com.starrocks.statistic.AnalyzeMgr.clearStatisticFromDroppedPartition(AnalyzeMgr.java:557)
        at com.starrocks.statistic.StatisticsMetaManager.runAfterCatalogReady(StatisticsMetaManager.java:521)
        at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72)
        at com.starrocks.common.util.Daemon.run(Daemon.java:98)

@Rocky

Rocky

04/13/2026, 2:50 PM

The log you shared is from yesterday (
2026-04-12
). It shows a failure in a background statistics thread, which is likely a symptom of the metadata being in an inconsistent state during the crash, but it is not the reason your FE is currently refusing to start. Based on your previous message from today (
2026-04-13
), the real blocker is that the FE is stuck in a loop trying to connect to itself as a helper node (

127.0.0.1:9010

) and failing. How to get the FE "Alive" (Port 9030) Since you are in a recovery scenario (single pod, shared-data mode, IP likely changed), follow these steps to force the FE to stabilize: 1. Fix the Configuration The FE is currently stuck because it doesn't realize it should be the Leader. Edit

/opt/starrocks/fe/conf/fe.conf

and add these recovery flags:

Copy code

properties
# Force the FE to ignore previous IP/group history and become Leader
bdbje_reset_election_group = true

# Force the StarOS (shared-data) manager to reset its election
starmgr_bdbje_reset_election_group = true

2. Start the FE correctly Do not use the

--helper

flag pointing to

127.0.0.1

. Start it normally so it can read the reset flags from the config:

Copy code

bash
# Kill any hung FE processes first
ps ux | grep StarRocksFE | grep -v grep | awk '{print $2}' | xargs kill -9 2>/dev/null

# Start the FE
/opt/starrocks/fe/bin/start_fe.sh --daemon

3. Verify the CURRENT log Watch the current log file (not the

.20260412

one):

Copy code

bash
tail -f /opt/starrocks/fe/log/fe.log

Look for this specific line:

[TransferService.run():...] transfer from UNKNOWN to LEADER

Once you see "transfer to LEADER", the FE will open port 9030 and the "not alive" error will disappear. 4. Important: Cleanup Once the FE is healthy (you can log in via MySQL): 1. Remove those two lines from

fe.conf

. 2. If you don't remove them, the cluster will reset its metadata every time the pod restarts, which will cause data consistency issues later. Regarding the
IllegalStateException
in your log: This error in

statistics meta manager

happens when StarRocks tries to clean up statistics for a table or partition that is already in a "dropped" or "invalid" state in the metadata. * Is it fatal? No, it's a daemon thread. It will log the error and try again later. * Why did it happen? It usually happens if the metadata journals were interrupted during a drop operation. * How to fix? Once the FE is back online as a Leader, this background task will usually resolve itself or can be fixed by running

ANALYZE TABLE <table_name>

to refresh the stats. Focus on Step 1 and 2 first to get the service listening on port 9030. References • knowledge/trouble_shooting/fe_deadlock_problems.md • data_source/catalog/iceberg/iceberg_meta_table.md • data_source/catalog/iceberg/DDL.md • sql-reference/information_schema/partitions.md

Abhishek Yadav

04/13/2026, 5:17 PM

Copy code

/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2026-04-14T01:17:09.204+0800	ERROR	StarRocksClusterReconciler	sub controller reconciles spec failed	{"name": "kube-starrocks", "namespace": "starrocks", "subController": "feController", "error": "the replicas of statefulset kube-starrocks-fe can not be scaled to 1"}
<http://github.com/StarRocks/starrocks-kubernetes-operator/pkg/controllers.(*StarRocksClusterReconciler).Reconcile|github.com/StarRocks/starrocks-kubernetes-operator/pkg/controllers.(*StarRocksClusterReconciler).Reconcile>
	/go/src/app/pkg/controllers/starrockscluster_controller.go:94
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2026-04-14T01:17:09.247+0800	ERROR	Reconciler error	{"controller": "starrockscluster", "controllerGroup": "<http://starrocks.com|starrocks.com>", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"starrocks"}, "namespace": "starrocks", "name": "kube-starrocks", "reconcileID": "3cef766b-f94c-4278-aa3c-9177f0c7d34a", "error": "the replicas of statefulset kube-starrocks-fe can not be scaled to 1"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler>

@Rocky getting this error in operator pod

Rocky

04/13/2026, 5:18 PM

This error in the StarRocks Operator occurs because the Operator has built-in safety checks for Scaling FE nodes. The message

the replicas of statefulset kube-starrocks-fe can not be scaled to 1

means the Operator is refusing to scale your FE cluster down to a single node. This usually happens for one of two reasons: 1. Safety Check (No Leader) The Operator will not allow scaling operations (especially scaling down) if it cannot find a Healthy Leader. Since your FE service is "not alive yet," the Operator cannot connect to the cluster to safely run the

ALTER SYSTEM DROP FOLLOWER

commands required to remove nodes from the BDBJE group. It blocks the scale-down to prevent permanent metadata corruption or quorum loss. 2. Quorum Protection StarRocks FEs require a quorum (majority). If you have a 3-node or 5-node cluster and you try to jump directly to 1 node via the Kubernetes YAML, the Operator blocks it because simply deleting the pods would break the internal metadata election for the remaining node. ────────── How to Fix This Step 1: Revert the Replicas in your YAML To stop the Operator from looping on this error, set the

replicas

in your

StarRocksCluster

YAML back to the original number (e.g., 3). This allows the Operator to focus on just getting the existing pods to run rather than trying to delete them. Step 2: Fix the "Not Alive" FE (Manual Recovery) Since the Operator is stuck because the FE is not alive, you must manually "force" one FE pod to become the Leader. Follow these steps on the pod you want to be the Leader (usually

kube-starrocks-fe-0

or whichever has the most recent meta): 1. Edit the config inside the pod:

Copy code

bash
    vi /opt/starrocks/fe/conf/fe.conf

2. Add these recovery lines (at the very bottom):

Copy code

properties
    metadata_failure_recovery = true
    bdbje_reset_election_group = true
    # If using shared-data mode, also add:
    starmgr_bdbje_reset_election_group = true

3. Restart the FE process manually inside that pod:

Copy code

bash
    /opt/starrocks/fe/bin/stop_fe.sh
    /opt/starrocks/fe/bin/start_fe.sh --daemon

4. Verify the logs: Run

tail -f /opt/starrocks/fe/log/fe.log

. Look for the message:

transfer from UNKNOWN to LEADER

Step 3: Let the Operator Reconcile Once that one pod becomes the LEADER and Port 9030 is open, the Operator's error will disappear. It will see a healthy cluster and proceed with reconciliation. Step 4: Cleanup 1. Crucial: Once the FE is healthy, go back into

fe.conf

and remove the recovery lines you added in Step 2. 2. If you still want to scale down to 1 node, you can now change the

replicas: 1

in your YAML. Now that there is a Leader, the Operator can safely handle the scale-down process. Summary of the Situation The Operator error is a symptom, not the cause. It is "protecting" you from scaling a broken cluster. Focus on getting one FE pod into the

LEADER

state using the

bdbje_reset_election_group

flag, and the Operator will follow along once the service is alive. References • deployment/sr_operator.md • deployment/downgrade.md • administration/stargo.md • faq/Deploy_faq.md • administration/management/Scale_up_down.md

Abhishek Yadav

04/13/2026, 5:25 PM

Copy code

2026-04-14 01:25:20.443+08:00 INFO (statistics meta manager|28) [LocalMetastore.createDb():416] createDb dbName = _statistics_, id = 10001
2026-04-14 01:25:20.444+08:00 INFO (statistics meta manager|28) [StatisticsMetaManager.createDatabase():89] create statistics db down
2026-04-14 01:25:20.445+08:00 INFO (statistics meta manager|28) [StatisticsMetaManager.createSampleStatisticsTable():143] create sample statistics table start
2026-04-14 01:25:20.445+08:00 WARN (statistics meta manager|28) [StatisticsMetaManager.createSampleStatisticsTable():164] Failed to create sample statistics, 
com.starrocks.common.NoAliveBackendException: No alive backend
        at com.starrocks.common.util.AutoInferUtil.calDefaultReplicationNum(AutoInferUtil.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.statistic.StatisticsMetaManager.createSampleStatisticsTable(StatisticsMetaManager.java:147) ~[starrocks-fe.jar:?]
        at com.starrocks.statistic.StatisticsMetaManager.createTable(StatisticsMetaManager.java:402) ~[starrocks-fe.jar:?]
        at com.starrocks.statistic.StatisticsMetaManager.refreshStatisticsTable(StatisticsMetaManager.java:484) ~[starrocks-fe.jar:?]
        at com.starrocks.statistic.StatisticsMetaManager.runAfterCatalogReady(StatisticsMetaManager.java:514) ~[starrocks-fe.jar:?]
        at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72) ~[starrocks-fe.jar:?]
        at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?]
2026-04-14 01:25:20.446+08:00 WARN (statistics meta manager|28) [StatisticsMetaManager.refreshStatisticsTable():487] create statistics table table_statistic_v1 failed
2026-04-14 01:25:20.746+08:00 INFO (starrocks-mysql-nio I/O-4|140) [AcceptListener.handleEvent():80] Connection established. remote=/192.168.59.64:61461, connectionId=16777690

@Rocky

Rocky

04/13/2026, 5:25 PM

Great news! Your FE is finally alive and acting as the Leader. The logs you see now (

Connection established

and

create statistics db down

) indicate that the FE has successfully initialized its metadata and is ready to accept commands. The error

com.starrocks.common.NoAliveBackendException: No alive backend

is expected at this exact moment. It means the FE is trying to create the internal

_statistics_

tables, but it cannot find any active Backends (BE) or Compute Nodes (CN) to store the data. Why this is happening: 1. FE is Up: The FE process is now listening on port 9030 (MySQL). 2. BEs are Missing/Dead: Either the BE pods haven't started yet, or they haven't been added to this "newly reset" FE cluster. 3. Operator Sync: Since you were seeing Operator errors earlier, the Operator might still be waiting to sync the BE/CN state. ────────── Recommended Steps to Finish the Recovery 1. Check the Cluster Status via MySQL Now that the FE is responding, log into it using a MySQL client to see what it knows:

Copy code

bash
# From within the FE pod or a pod with mysql client
mysql -h 127.0.0.1 -P 9030 -u root

Once logged in, run:

Copy code

sql
SHOW BACKENDS;
-- OR if you are using shared-data mode (CNs)
SHOW COMPUTE NODES;

* If the list is empty: The FE doesn't know about any workers. You (or the Operator) need to add them. * If
Alive
is `false`: The workers are registered but the FE cannot reach them. Check the BE/CN logs. 2. Check the Operator Pod Check the logs of the

starrocks-operator

pod again. Now that the FE is alive, the error

the replicas of statefulset kube-starrocks-fe can not be scaled to 1

should disappear, and it should start automatically adding the BEs/CNs back to the cluster. 3. Manual Addition (If Operator is not doing it) If you are in an emergency and the Operator isn't adding the nodes, you can add them manually via MySQL:

Copy code

sql
-- Replace with the actual IP of your BE/CN pod
ALTER SYSTEM ADD BACKEND "192.168.x.x:9050"; 
-- OR for CNs
ALTER SYSTEM ADD COMPUTE NODE "192.168.x.x:9050";

4. Final Cleanup If you added

bdbje_reset_election_group = true

to your

fe.conf

to get it started: 1. Remove those lines from

fe.conf

now. 2. The FE is already the Leader; keeping those flags in the config will cause it to reset every time the pod restarts, which will break your cluster later. Summary You have passed the hardest part (FE metadata recovery). The

NoAliveBackendException

will go away automatically as soon as the first BE/CN pod successfully connects to the FE. Check your BE pod logs to ensure they aren't crashing or having network issues reaching the FE on port 9010. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • quick_start/shared-data.md • deployment/deploy_manually.md

Abhishek Yadav

04/13/2026, 5:45 PM

Copy code

root@kube-starrocks-fe-4:/opt/starrocks# tail -f /opt/starrocks/fe/log/fe.log
2026-04-14 01:44:47.754+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:44:49.755+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:44:51.755+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:44:53.756+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:44:55.756+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:44:55.757+08:00 WARN (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1232] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details.
2026-04-14 01:44:57.757+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:44:59.758+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:45:01.758+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:45:03.759+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2026-04-14 01:45:05.759+08:00 INFO (UNKNOWN 192.168.247.68_9010_1776091504458(-1)|1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: INIT. is ready: false

@Rocky how to fix this

Rocky

04/13/2026, 5:45 PM

The log message

wait globalStateMgr to be ready. FE type: INIT. is ready: false

indicates that this FE pod is stuck in the Initialization/Election phase. It cannot find a Leader to join, and it cannot elect itself as a Leader because it doesn't have a majority (quorum). Since you are running on Kubernetes and previously saw errors about "scaling to 1," you likely have a quorum loss (e.g., you had 3 or 5 FEs, but now only 1 is running or they cannot talk to each other). Why this is happening: * Quorum Loss: BDBJE (the metadata engine) requires a majority of nodes to be up. If you have 5 nodes (

fe-0

fe-4

) and only

fe-4

is starting, it will wait forever for the others. * Operator Conflict: The StarRocks Operator is blocking the scale-down to 1 because doing so manually without a healthy Leader can lead to permanent metadata split-brain. ────────── How to Fix (Recovery Steps) To force this FE to become the Leader and start the service, follow these steps: 1. Stop the Operator (Optional but Recommended) To prevent the Operator from restarting pods while you are fixing them, you can temporarily scale the operator deployment to 0:

Copy code

bash
kubectl scale deployment starrocks-operator -n <operator-namespace> --replicas=0

2. Force the FE to become Leader You need to tell this FE to ignore the other missing nodes and start as a standalone Leader. 1. Exec into the pod

kube-starrocks-fe-4

. 2. Edit the configuration file:

Copy code

bash
    vi /opt/starrocks/fe/conf/fe.conf

3. Add these lines at the end of the file:

Copy code

properties
    # Forcing this node to be the Leader
    bdbje_reset_election_group = true
    # If you are using shared-data mode, also add:
    starmgr_bdbje_reset_election_group = true

4. Restart the FE process inside the pod:

Copy code

bash
    /opt/starrocks/fe/bin/stop_fe.sh
    /opt/starrocks/fe/bin/start_fe.sh --daemon

3. Verify Recovery Watch the logs again:

tail -f /opt/starrocks/fe/log/fe.log

. You should see: *

transfer from UNKNOWN to LEADER

finish replay journals

* Port 9030 (MySQL) will finally open. 4. Clean Up and Resume Operator Once the FE is healthy: 1. Log in via MySQL and check the state:

SHOW FRONTENDS;

. 2. Remove the

bdbje_reset_election_group

lines you added to

fe.conf

. (If you leave them, the cluster will reset every time the pod restarts). 3. Scale the Operator back up:

Copy code

bash
    kubectl scale deployment starrocks-operator -n <operator-namespace> --replicas=1

4. Update your

StarRocksCluster

YAML to match the actual number of FE pods you want to keep. The Operator error about "scaling to 1" should resolve now that it can talk to a healthy Leader. Summary Your FE is "waiting for friends" that aren't coming. Using

bdbje_reset_election_group

tells the FE: "I am the only one left, make me the Leader." References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/14/2026, 5:26 AM

Copy code

[NodeMgr.getHelperNodes():656] get helper nodes: [192.168.6.137:9010]
2026-04-14 02:41:50.275+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():508] Current run_mode is shared_data
2026-04-14 02:41:50.275+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():513] Got role: LEADER, node name: 192.168.124.76_9010_1776088663466 and run_mode: shared_data

@Rocky

Rocky

04/14/2026, 5:27 AM

Your FE has successfully transitioned from the

INIT

state to the LEADER
state! This means the metadata recovery or election was successful, and the FE process is now the master of the cluster. Specifically, the log line

Got role: LEADER, node name: 192.168.124.76_9010...

confirms that this node is now running as the Leader in

shared_data

(Cloud Native) mode. What to do next: 1. Verify via MySQL The FE should now be listening on the MySQL port (default

). Log in to verify the cluster state:

Copy code

bash
# Connect to the FE (use the Pod IP or service)
mysql -h 127.0.0.1 -P 9030 -u root

Once connected, run these commands:

Copy code

sql
-- Check if the FE is healthy and shows as LEADER
SHOW FRONTENDS\G

-- Check if Compute Nodes (CNs) have joined (since you are in shared_data mode)
SHOW COMPUTE NODES;

2. Address "No Alive Backend" (if it persists) If

SHOW COMPUTE NODES

(or

SHOW BACKENDS

) is empty, the FE is up but has no workers to execute queries. * Now that the FE is a Leader, the StarRocks Operator should automatically detect the healthy FE and begin registering the CN/BE pods to it. * If they don't appear after a few minutes, check the logs of your

be

cn

pods to see if they are failing to connect to the new FE Leader IP (

192.168.124.76

). 3. Important Cleanup If you added the recovery flags to

fe.conf

(like

bdbje_reset_election_group = true

) to force the Leader election: 1. Remove those lines from
fe.conf
now. 2. If you leave them in, the FE will try to "reset" itself every time the pod restarts, which will cause metadata inconsistencies and potentially break the cluster once you add more FE nodes (Followers). 4. Resolve the Operator Scaling Error Now that there is a healthy Leader, the Operator's error (

the replicas of statefulset kube-starrocks-fe can not be scaled to 1

) should resolve itself. The Operator was blocking the scale-down because it couldn't find a Leader to safely execute the decommissioning of the other nodes. If the Operator is still stuck, you may need to: 1. Ensure the

StarRocksCluster

CRD

replicas

matches the actual number of pods you want. 2. If you scaled the operator to 0 earlier, scale it back up to 1:

kubectl scale deployment starrocks-operator -n <namespace> --replicas=1

Summary The

Got role: LEADER

log is the "all-clear" signal for the FE. Your next focus is ensuring the Compute Nodes (Backends) connect to this Leader and then cleaning up your

fe.conf

to ensure future stability. References • deployment/sr_operator.md • quick_start/helm.md • administration/stargo.md • deployment/helm.md

Abhishek Yadav

04/14/2026, 5:51 AM

@Rocky explain fe/meta# ls bdb image lost+found what these folder keeps

Rocky

04/14/2026, 5:51 AM

The

fe/meta

directory is the most critical part of your StarRocks installation. It stores all the metadata for your cluster (databases, tables, permissions, configurations, and the state of your backends). Here is what each folder inside

fe/meta

keeps: 1.
bdb
(Berkeley DB Java Edition) This folder contains the Journal Logs (Write-Ahead Logs) for your metadata. * What it does: Every time you run a command like

CREATE TABLE

ALTER USER

, the FE records that change as a "journal" entry in these BDB files before applying it. * Why it exists: In a multi-FE cluster, BDBJE is used to synchronize metadata between the Leader and the Followers. If an FE crashes, it reads these logs upon restart to "replay" all actions and bring itself back to the current state. * Key Files: You will see files named like

00000000.jdb

. Never delete or modify these manually unless performing emergency recovery. 2.
image
This folder contains Metadata Snapshots (Checkpoints). * What it does: Periodically, the FE's "Checkpoint" thread takes all the journals in the

bdb

folder and "flattens" them into a single image file (representing the current total state of the system). * Why it exists: It prevents the FE from having to replay millions of logs from the beginning of time. When an FE starts, it loads the latest

image.<version>

file first, and then only replays the few BDB journals that were created after that image was saved. * Key Files:

image.<version>

and

ROLE

(which tells the FE if it was a Leader, Follower, or Observer during the last run). 3.
lost+found
This is a standard Linux filesystem artifact. * What it does: It is created by the Linux OS (usually on ext3/ext4 filesystems) to store recovered file fragments after a filesystem check (

fsck

) following a crash. * Why it is there: If your

meta_dir

is a dedicated disk mount, Linux automatically creates this. It is not a StarRocks-specific folder and is usually empty. You can safely ignore it. ────────── Summary Table Folder | Content | Importance bdb
| Transaction/Journal Logs | Critical. Used for replaying recent changes and sync. image
| Metadata Snapshots | Critical. Used for fast startup and base state. lost+found
| System recovery bits | Low. OS-level folder; usually empty. Rocky's Pro-Tip: Always back up the

fe/meta

directory before performing major upgrades or manual recovery steps. If you lose both the

image

and

bdb

folders, your cluster metadata is gone! References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/14/2026, 5:55 AM

Copy code

root@kube-starrocks-fe-0:/opt/starrocks/fe/meta/image# cat ROLE
#Mon Apr 13 21:57:43 CST 2026
role=LEADER
hostType=
name=192.168.124.76_9010_1776088663466

@Rocky explain what this means

Rocky

04/14/2026, 5:55 AM

The

ROLE

file in

fe/meta/image/

is a small but critical text file that StarRocks uses to remember its identity and status across restarts. It prevents the node from getting "confused" about who it is when the process starts up. Here is a breakdown of your specific output:

Copy code

properties
#Mon Apr 13 21:57:43 CST 2026
role=LEADER
hostType=
name=192.168.124.76_9010_1776088663466

1.
role=LEADER
This indicates that the last time this FE was running, it was the Leader of the cluster. * Leader: The only node that can write metadata (e.g., create tables, load data). * Follower: A node that can be elected as Leader and maintains a full copy of metadata. * Observer: A read-only node used to scale query concurrency (cannot become a Leader). 2.
hostType=
This field specifies how the node identifies its address. * In your case, it is empty, which usually means the node is using the default IP-based identification. * If you were using Fully Qualified Domain Names, this would typically say

hostType=FQDN

. 3.
name=192.168.124.76_9010_1776088663466
This is the Unique Identifier for this specific FE node in the metadata. It follows a strict format: * 192.168.124.76
: The IP address the node is bound to. * 9010
: The

edit_log_port

(used for internal BDBJE communication). * 1776088663466
: A unique timestamp (Unix epoch in ms) generated when the FE was first initialized. ────────── Why this file matters: 1. Identity Protection: If you try to start an FE with a different IP than what is recorded in the

name

field (without using FQDN), the FE will refuse to start to prevent metadata corruption. 2. Leader Persistence: When the cluster restarts, the FE reads this file to know if it should immediately try to resume its duties as a Leader or if it should wait to find a Leader as a Follower. 3. Recovery: If you ever need to perform a manual recovery (like the

bdbje_reset_election_group

fix we discussed), StarRocks updates this file once the node successfully takes over as the new Leader. Rocky's Note: You should never manually edit this file unless you are performing an advanced metadata recovery. If this file is deleted, the FE will think it is a brand-new node and will fail to join the existing cluster because it won't recognize its own history. References • administration/Meta_recovery.md • quick_start/shared-nothing.md • quick_start/helm.md • administration/management/FE_configuration.md • introduction/Architecture.md

Abhishek Yadav

04/14/2026, 5:57 AM

@Rocky if fe pods are not able to come up - should we clear bdb to atleast restore it to some point ?

Rocky

04/14/2026, 5:57 AM

Clearing the

bdb

folder is a powerful but dangerous action. Whether you should do it depends entirely on whether you have a working "Leader" node in your cluster. Scenario A: You have a healthy Leader running Yes, you can (and often should) clear the
meta_dir
or
bdb
folder. If one of your Follower FE pods is stuck and won't start due to BDBJE corruption (e.g.,

VLSN

errors in the logs), the standard fix is: 1. Stop the broken FE pod. 2. Delete the entire
meta/
directory on that specific node. 3. Restart the FE pod with the

--helper

flag pointing to the healthy Leader. _ _Result:* The FE will start "empty," connect to the Leader, and download a fresh, healthy copy of the metadata. No data is lost. ────────── Scenario B: The entire cluster is down (No Leader) Do NOT clear
bdb
yet. This is a last resort. If your FE cannot start and you have no other working FE to act as a helper: * If you clear
bdb
but keep `image`: The FE will start using the last snapshot. Any data loaded or tables created after that snapshot was taken will be permanently lost. * If you clear both
bdb
and `image`: You have deleted your entire cluster's metadata. Your databases and tables will be "gone" as far as StarRocks is concerned. What to do instead for Scenario B: Before deleting files, try these "Safe" recovery steps in order: 1. Check for Disk Space: StarRocks FEs will refuse to start if the disk has less than 5GB of free space. 2. Force a New Leader (Metadata Recovery Mode): If the FE is stuck in an election loop, add this to `fe.conf`:

Copy code

properties
    metadata_failure_recovery = true

Restart the FE. This forces the node to ignore the "quorum" and promote itself to Leader. Once it starts, remove this line immediately from
fe.conf
. 3. Check for IP Changes: If you are on Kubernetes and your Pod IP changed, but you aren't using FQDNs, the FE will fail because the IP in the

meta/image/ROLE

file doesn't match. When is clearing
bdb
the "Right" move? Only when the logs explicitly show a corrupted BDB log file (e.g.,

ChecksumBackendException

LogFileNotFoundException

) and you have a backup or a working Leader. Rocky's Recommendation: If you are currently stuck, check your
fe.log
first. * If you see

REPLICA_CANNOT_BE_PROMOTED

, use

metadata_failure_recovery = true

. * If you see

vlsn

errors, and you have a Leader, wipe the

meta/

folder and restart. * Always back up the
meta/
folder to a

.zip

file before you delete anything! References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/sr_operator.md • quick_start/shared-nothing.md References • administration/Meta_recovery.md

Abhishek Yadav

04/14/2026, 6:02 AM

@Rocky the bdb folder have 2 type of file 1 with .jdb extension and another with .csv or je.info.x type... ideall what we should delete to recover it

Rocky

04/14/2026, 6:02 AM

In the

fe/meta/bdb

directory, you are seeing the internal files of Berkeley DB Java Edition (BDBJE), which StarRocks uses to store its metadata journals. To answer your question directly: Ideally, you should not delete individual files manually to "fix" a node. Instead, you should clear the entire directory if you have a helper, or use specific configuration flags if you don't. Here is the breakdown of those files and what to do with them: 1.
.jdb
Files (e.g.,
00000000.jdb
) * What they are: These are the Journal Data Blocks. They contain the actual metadata (your table schemas, load jobs, user permissions, etc.). * Can you delete them? * No, if this is your only FE node or the Leader. Deleting these is equivalent to deleting your database's memory. * Yes, if you have another healthy FE (Leader) running. In that case, you delete the entire

meta

folder so the node can pull a fresh, clean copy from the Leader. 2.
je.info.x
Files (e.g.,
je.info.0
,
je.info.1
) * What they are: These are internal log files generated by the BDBJE library. They contain trace information and error messages about the BDB engine itself (different from StarRocks'

fe.log

). * Can you delete them? * Yes. These are just logs. Deleting them is safe and will not cause data loss, but it also won't fix a "Metadata Damage" or "Checksum" error because the actual data is in the

.jdb

files. 3.
.csv
Files (e.g.,
je.stat.csv
) * What they are: These are Statistics Files. They are usually created if BDBJE statistics collection is enabled. They track performance metrics like cache hits and cleaning progress. * Can you delete them? * Yes. These are purely for monitoring/debugging. Deleting them is 100% safe. ────────── What should you delete to recover? Scenario A: You have a healthy Leader FE running If this pod is a "Follower" that won't start: 1. Stop the pod. 2. Delete everything inside

fe/meta/

(including the

image

and

bdb

folders). 3. Restart the pod using the

--helper

flag pointing to the Leader. _ _Example:*

./bin/start_fe.sh --helper <leader_ip>:9010 --daemon

* This is the most reliable way to recover a broken node. Scenario B: All FE nodes are down (No Leader) If the files are corrupted and you have no other node to sync from: 1. Do NOT delete the
.jdb
files. If you delete them, your metadata is gone. 2. Try to force a recovery by adding this to `fe.conf`:

Copy code

properties
    metadata_failure_recovery = true

3. Restart the FE. If it starts, it will become the new Leader. 4. Important: Once it is up, immediately remove that line from

fe.conf

and restart it again normally. Rocky's Warning: Before you delete or modify anything in

fe/meta

, always make a backup (e.g.,

cp -r meta meta_backup

). If you accidentally delete the wrong

.jdb

file without a backup, you may lose your entire cluster configuration. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/Others.md

Abhishek Yadav

04/14/2026, 6:03 AM

@Rocky we don't have any healthy fe pod running, we do know pod which was leader before, also tried using metadata_failure_recovery it did not work..

Rocky

04/14/2026, 6:03 AM

metadata_failure_recovery = true

did not work and no FE pods are coming up, you are likely facing one of two issues: BDBJE Election Group Corruption or a Kubernetes Pod IP Mismatch. When the entire cluster is down, you must manually force one node to become the "Seed" Leader again. Follow these steps precisely: Step 1: Isolate a Single Pod In Kubernetes, the FE pods will keep trying to find each other and failing. You need to stop the noise. 1. Scale your FE StatefulSet down to 1 replica (choose the one that was the Leader, e.g.,

fe-0

Copy code

bash
    kubectl scale statefulset <fe-statefulset-name> --replicas=1

2. Wait for the other FE pods to terminate. Step 2: Use the "Stronger" Recovery Flag The

metadata_failure_recovery

flag is older. In recent versions, it has been replaced by a more powerful flag. 1. Update your FE

ConfigMap

(or the

starrocksCluster

CRD if using the Operator) to include:

Copy code

properties
    bdbje_reset_election_group = true

2. Delete the remaining FE pod so it restarts with this new config.

Copy code

bash
    kubectl delete pod kube-starrocks-fe-0

Step 3: Check for IP Mismatch (The K8s Trap) Since your

ROLE

file showed an IP (

192.168.124.76

), StarRocks is using IP-based identification. If your pod restarted and received a new IP, BDBJE will refuse to start because it thinks it is a different machine. 1. Check the current IP of your pod:

Copy code

bash
    kubectl get pod kube-starrocks-fe-0 -o wide

2. Compare it to the IP in

/opt/starrocks/fe/meta/image/ROLE

. 3. If they don't match: You must manually update the

ROLE

file to reflect the new IP. _ _Warning:* This is risky. Back up the file first. * Edit the

name=

line in the

ROLE

file to replace the old IP with the new pod IP, keeping the port and timestamp the same. * Restart the pod. Step 4: Verify and Clean Up 1. Monitor the logs:

kubectl logs -f kube-starrocks-fe-0

. 2. If the FE starts successfully, it will log:

transfer to leader

. 3. Immediately remove

bdbje_reset_election_group = true

from your config and restart the pod once more. If you leave it in, it can cause issues during future scale-outs. 4. Once the first FE is healthy, scale the StatefulSet back up to 3 replicas. The other 2 pods should join as followers automatically. Summary of what to look for in `fe.log`: * REPLICA_CANNOT_BE_PROMOTED
: Means it's waiting for a majority that doesn't exist.

bdbje_reset_election_group

fixes this. * HANDSHAKE_ERROR
: Usually means an IP or Port mismatch between what's in the metadata and the current environment. * insufficient log files
: Your

bdb/

files are corrupted. You may need to restore from a physical backup of the

meta/

directory. Rocky's Pro-Tip: To prevent this from happening again on Kubernetes, enable FQDN (Fully Qualified Domain Names) in your

fe.conf

(

enable_fqdn_mode = true

). This allows StarRocks to use the pod's network name (which never changes) instead of its IP (which changes every restart). References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/sr_operator.md • administration/management/FE_FQDN.md References • administration/Meta_recovery.md

Abhishek Yadav

04/14/2026, 6:05 AM

Copy code

we have tried metadata_failure_recovery=true
bdbje_reset_election_group = true
starmgr_bdbje_reset_election_group = true

nothing worked @Rocky

Rocky

04/14/2026, 6:05 AM

If the standard flags (

metadata_failure_recovery

bdbje_reset_election_group

, etc.) have already failed, you are likely dealing with physical corruption of the BDB logs or a node identity conflict (common in Kubernetes). At this point, you have two options: The Last Resort Recovery (start from image) or Checking for IP/Network Lock. Option 1: The "Last Resort" Recovery (Start from Image) If the BDB logs are so corrupted that they cannot be replayed or reset, you can force the FE to start from its last successful snapshot (the

image

file). Note: You will lose any metadata changes (table creations, schema updates, load jobs) that happened after the last image was saved. 1. Stop all FE pods (scale the StatefulSet to 0). 2. Take a physical backup of the

meta/

directory on the node you believe was the most recent Leader. 3. Perform the "Wipe": * Navigate to your

meta_dir

(usually

/opt/starrocks/fe/meta

). * Delete the entire
bdb/
folder. * Keep the
image/
folder. 4. Configure for Bootstrap: * Ensure

fe.conf

has ONLY:

bdbje_reset_election_group = true

. * (Remove

metadata_failure_recovery

and

starmgr_bdbje_reset_election_group

for now). 5. Restart one FE pod. * Scale the StatefulSet to 1. * If it starts, it will read the

image

, see that there are no BDB logs, and create a brand new election group starting from that image. 6. Verify: Run

SHOW FRONTENDS;

. If the node is

LEADER

and

Alive: true

, remove the flag from

fe.conf

and scale up your other pods. They will sync from this new leader. ────────── Option 2: Check for IP/Name Mismatch (The K8s Trap) If you see errors like

HANDSHAKE_ERROR

Node already exists

in the

fe.log

, BDBJE is rejecting the pod because its current IP/Hostname doesn't match what is recorded in the metadata. 1. Check your

fe/meta/image/ROLE

file. It contains the "Name" of the node (usually an IP and timestamp). 2. Compare that IP to the current Pod IP (

kubectl get pod -o wide

). 3. If they don't match: * BDBJE thinks the "old" pod is still part of the group and the "new" pod is an intruder. * In this case, clearing the

bdb

folder and keeping the

image

(as described in Option 1) is often the only way to "forget" the old IP and start fresh. ────────── Critical: Check the
fe.log
for these specific errors To give you a better direction, search your

fe.log

for these keywords: * ChecksumException
or
LogFileNotFoundException
: Your BDB files are physically broken. Use Option 1. * INSUFFICIENT_LOGS
: The node is too far behind the group. If all nodes say this, you must use Option 1. * REPLICA_CANNOT_BE_PROMOTED
: This usually means the node doesn't think it's allowed to be leader.

bdbje_reset_election_group = true

should fix this, provided you have only one FE running while you try to recover. Rocky's Advice: If you have multiple FE pods, scale down to 1 FE pod before trying any of these steps. Trying to recover a 3-node cluster while all 3 are fighting to become leader will often cause recovery flags to fail. References • administration/Meta_recovery.md • faq/Others.md • administration/management/FE_configuration.md • administration/management/FE_FQDN.md References • administration/Meta_recovery.md • faq/Others.md • administration/management/FE_configuration.md • administration/management/FE_FQDN.md References

Rocky

04/14/2026, 6:05 AM

• administration/Meta_recovery.md

Abhishek Yadav

04/14/2026, 6:18 AM

@Rocky why are we getting

2026-04-14 14:14:05.252+08:00 ERROR (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():498] Unmatched run mode between config file and version file: shared_nothing vs shared_data. will exit!

even fe.conf have run_mode as shared_data

Rocky

04/14/2026, 6:18 AM

This error occurs because there is a fundamental mismatch between the configuration in your

fe.conf

and the existing metadata stored in your

meta

directory. In StarRocks, the

run_mode

(Shared-Nothing vs. Shared-Data) is locked in the moment the cluster is first initialized. You cannot switch a cluster from one mode to the other after it has been created. What the error means: * shared_nothing
(from Version File): Your existing metadata (located in

fe/meta/image/VERSION

) was created for a standard Shared-Nothing cluster. * shared_data
(from Config File): Your current

fe.conf

is telling the FE to start in Shared-Data mode. Why is this happening? 1. Accidental Change: You might have manually added

run_mode = shared_data

to your

fe.conf

during the recovery process, but your cluster was originally a standard Shared-Nothing cluster. 2. Reused Volume: If you are on Kubernetes, your Pod might be mounting a Persistent Volume (PV) that contains old metadata from a previous (Shared-Nothing) installation. 3. Incompatible Image: You are trying to recover using a metadata backup/image that was taken from a Shared-Nothing cluster. ────────── How to Fix It Case A: Your cluster is actually Shared-Nothing (Most Common) If your cluster uses local disks on BEs to store data, it is a Shared-Nothing cluster. 1. Open your

fe.conf

. 2. Change

run_mode

shared_nothing

(or simply remove the line, as

shared_nothing

is the default). 3. Restart the FE. Case B: You intended to use Shared-Data (Cloud-Native) If you are trying to set up a new Shared-Data cluster (using S3/MinIO/Azure Storage): 1. You cannot reuse the old metadata. The error exists because the FE sees "Shared-Nothing" metadata and refuses to corrupt it with "Shared-Data" operations. 2. If this is a fresh setup and you don't care about the existing data: Wipe the
meta
directory completely (

rm -rf /path/to/fe/meta/*

). 3. Ensure your

fe.conf

has all the required Shared-Data parameters (

run_mode

cloud_native_storage_type

aws_s3_path

, etc.). 4. Restart the FE to initialize a fresh Shared-Data cluster. How to verify the Metadata Mode You can check what the metadata thinks it is by looking at the

VERSION

file:

Copy code

bash
cat /path/to/fe/meta/image/VERSION | grep run_mode

If it says

run_mode=shared_nothing

, then your

fe.conf

must match that, or you must delete the metadata to start over. Rocky's Note: Since you were already in a recovery scenario, it is highly likely that your cluster was Shared-Nothing all along. Try removing

run_mode = shared_data

from your config and see if the FE moves past this specific to the next step of the BDBJE election stage. References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md

Abhishek Yadav

04/14/2026, 6:19 AM

Copy code

root@kube-starrocks-fe-0:/opt/starrocks# cat fe/meta/image/VERSION
#Mon Apr 13 18:24:24 CST 2026
clusterId=1793171692
runMode=shared_data

@Rocky but even version haev shared_Data mentioned

Rocky

04/14/2026, 6:20 AM

If the

VERSION

file says

runMode=shared_data

but you are getting the error

shared_nothing vs shared_data

, it means the FE process thinks its configuration is set to

shared_nothing

(which is the default). Since you are in a Kubernetes pod (

kube-starrocks-fe-0

), here are the most likely reasons why your

run_mode = shared_data

setting is being ignored: 1. The "ConfigMap" Trap (Most Likely) In Kubernetes, the

fe.conf

file is usually mounted from a ConfigMap. * If you manually edited

/opt/starrocks/fe/conf/fe.conf

inside the pod, those changes are not permanent. * When the pod restarts (which it does when it crashes with this error), it reverts to whatever is in the ConfigMap. * The Fix: You must edit the ConfigMap or the

StarRocksCluster

custom resource (if using the Operator) to include

run_mode = shared_data

. 2. Environment Variable Override The StarRocks startup script often checks for environment variables. Check if your Pod spec has an environment variable named

RUN_MODE

or similar. * Run:

kubectl get pod kube-starrocks-fe-0 -o yaml | grep -i run_mode

* If an environment variable is set to

shared_nothing

, it will override your

fe.conf

file. 3. Case Sensitivity & Typos StarRocks configuration is strict. Check your

fe.conf

for: * Correct Name: It must be

run_mode

(all lowercase, with an underscore). * Hidden Characters: Ensure there are no weird characters, trailing spaces, or BOM (Byte Order Mark) at the start of the file. * Duplicate Entries: Ensure

run_mode

isn't defined twice in the file (the last one wins). 4. Verify what the FE actually sees Before the FE crashes, it usually prints the configuration it loaded to the metadata log or

fe.out

. However, a quick way to check the file content as the FE sees it is:

Copy code

bash
kubectl exec kube-starrocks-fe-0 -- cat /opt/starrocks/fe/conf/fe.conf | grep run_mode

If this returns

shared_nothing

or nothing at all, your configuration update didn't "stick." How to Resolve: 1. If using the StarRocks Operator: Update your YAML file:

Copy code

yaml
    spec:
      starRocksFeSpec:
        configMapInfo:
          configMapName: fe-config
          resolveKey: fe.conf

And ensure that the ConfigMap

fe-config

contains

run_mode = shared_data

. 2. If you are trying to recover and nothing works: You can try to "force" the run mode by passing it as a flag in the start command (if your entrypoint allows it), but the best way is to fix the ConfigMap. Why the order matters: The error

shared_nothing vs shared_data

literally reads as

[What I am configured as] vs [What the disk says]

. * What I am configured as:

shared_nothing

(Default) * What the disk says:

shared_data

(from your

VERSION

file) Correct your configuration source (ConfigMap/Operator CRD), and the FE will start. References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md References

Rocky

04/14/2026, 6:20 AM

• administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md

Abhishek Yadav

04/14/2026, 6:22 AM

Copy code

containers:
        - name: fe
          image: starrocks/fe-ubuntu:3.5-latest
          command:
            - /bin/bash
            - '-c'
          args:
            - |
              echo "DEBUG MODE ENABLED - FE paused"
              sleep 3600

@Rocky is it because I added sleep cmd for debug as pods were crashing continuously. Previously it was using

Copy code

containers:
        - name: fe
          image: starrocks/fe-ubuntu:3.5-latest
          command:
            - /opt/starrocks/fe_entrypoint.sh
          args:
            - $(FE_SERVICE_NAME)

Rocky

04/14/2026, 6:23 AM

Yes, that is exactly why you are seeing this error. By changing the

command

and

args

to a

sleep

loop, you have bypassed the fe_entrypoint.sh
script. In StarRocks Kubernetes deployments (especially those managed by the Operator or Helm charts), that script is responsible for more than just starting the process. Why the error occurs now: 1. Config Injection: The

fe_entrypoint.sh

script typically reads Environment Variables (like those defined in your Kubernetes YAML) and injects them into the

fe/conf/fe.conf

file before starting the FE. 2. Default Value: Without that script running, the FE uses the default

fe.conf

packaged in the Docker image. In that default file,

run_mode

is either not set or set to shared_nothing
. 3. The Conflict: When you manually try to start the FE inside your debug shell, it reads the default

shared_nothing

config, but then it looks at your Persistent Volume's

meta/image/VERSION

file, which says

runMode=shared_data

. It sees the mismatch and crashes. How to fix it for debugging: If you need to keep the pod in a

sleep

state for debugging but still want to be able to start the FE manually, you have two options: Option 1: Manually update the config before starting Once you are inside the pod via `kubectl exec`: 1. Open the config file:

vi /opt/starrocks/fe/conf/fe.conf

2. Add (or change) the line:

run_mode = shared_data

3. (Crucial) Add your other shared-data configs as well (e.g.,

aws_s3_path

cloud_native_storage_type

, etc.), otherwise it will fail for other reasons. 4. Run the start command:

/opt/starrocks/fe/bin/start_fe.sh

Option 2: Run the Entrypoint script manually Instead of calling

bin/start_fe.sh

directly, try running the actual entrypoint with the expected arguments so it can perform the setup:

Copy code

bash
# Inside the pod
/opt/starrocks/fe_entrypoint.sh kube-starrocks-fe

(Note: Replace
kube-starrocks-fe
with your actual service name if different). Recommendation for "Continuous Crashing" Since you mentioned the pods were crashing continuously before you added the sleep: • The original crash was likely the real issue (perhaps BDBJE election failure or S3 connectivity). • Now that you have the sleep in place, check the

/opt/starrocks/fe/log/fe.log

and

/opt/starrocks/fe/log/fe.out

after you try to start it manually. The "Unmatched run mode" is a side effect of your debug method, not the original cause of your cluster failure. Rocky's Tip: Once you manually add

run_mode = shared_data

fe.conf

, you will likely get past this error and see the original error that caused the crash in the first place! References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md References • administration/management/FE_configuration.md • deployment/shared_data/azure.md • quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md References • administration/management/FE_configuration.md • deployment/shared_data/azure.md

Rocky

04/14/2026, 6:23 AM

• quick_start/shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md the first place. References • administration/management/FE_configuration.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md

Abhishek Yadav

04/14/2026, 6:26 AM

Copy code

root@kube-starrocks-fe-0:/opt/starrocks# vi /opt/starrocks/fe/conf/fe.conf
root@kube-starrocks-fe-0:/opt/starrocks# /opt/starrocks/fe/bin/start_fe.sh
root@kube-starrocks-fe-0:/opt/starrocks# tail -f /opt/starrocks/fe/log/fe.log
2026-04-14 14:25:43.122+08:00 ERROR (main|1) [StarRocksFE.start():207] StarRocksFE start failed
com.starrocks.journal.JournalException: Current node is not in the electable_nodes list. will exit
        at com.starrocks.journal.bdbje.BDBEnvironment.initConfigs(BDBEnvironment.java:182) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:172) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:151) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1211) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1182) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:143) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:93) ~[starrocks-fe.jar:?]

@Rocky what this means and how to fix it

Rocky

04/14/2026, 6:26 AM

The error

Current node is not in the electable_nodes list

means that the FE process is attempting to start, but the identity of the current pod (its IP or Hostname) does not match the list of "Follower" nodes stored in the existing metadata. In StarRocks, the metadata (BDBJE) keeps a hardcoded list of nodes allowed to participate in elections. If your Kubernetes Pod IP changed, or if you are trying to recover metadata on a new pod that has a different identity than the one that created the metadata, BDBJE will reject it to prevent cluster split-brain. How to Fix It Since you are already in a manual recovery/debug state, you need to force this FE to "reset" the election group and declare itself the new leader of this metadata. 1. Update
fe.conf
Add the following line to your

/opt/starrocks/fe/conf/fe.conf

file:

Copy code

properties
bdbje_reset_election_group = true

2. Start the FE Run the start script again:

Copy code

bash
/opt/starrocks/fe/bin/start_fe.sh

3. Verify Startup Check the logs again. The FE should now bypass the election check, start up, and promote itself to LEADER.

Copy code

bash
tail -f /opt/starrocks/fe/log/fe.log

Look for a log line like:

become leader...

transfer from UNKNOWN to LEADER

. 4. Clean up (Crucial) Once the FE is successfully running and you can connect to it via MySQL (

mysql -P9030

): 1. Check the frontend status:

SHOW FRONTENDS;

(The

Role

should be

LEADER

). 2. Stop the FE. 3. Remove the

bdbje_reset_election_group = true

line from

fe.conf

. 4. Restart the FE normally. ────────── Why did this happen? * Pod IP Change: If you are not using FQDNs for your FE nodes, K8s Pod IP changes cause the BDBJE election group (which is IP-based) to become invalid. * Manual Recovery: When you manually moved metadata or started the FE without the official entrypoint, the FE's current network identity didn't match what was written in the

meta/bdb

logs. Rocky's Pro-Tip: Once you get the FE running as LEADER, if you have other FE pods, you will likely need to clear their

meta

directories and re-add them as followers using:

ALTER SYSTEM ADD FOLLOWER "IP:9010";

And then start them with the

--helper

flag pointing to this leader. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • administration/Meta_recovery.md

Abhishek Yadav

04/14/2026, 6:28 AM

Copy code

2026-04-14 14:25:42.405+08:00 INFO (main|1) [StarRocksFE.start():137] StarRocks FE starting, version: 3.5.15-5abb1cb
2026-04-14 14:25:42.410+08:00 INFO (main|1) [NetUtils.getHosts():79] ipv6 link local address fe80:0:0:0:e893:ecff:fee0:6625%eth0 is skipped
2026-04-14 14:25:42.411+08:00 INFO (main|1) [FrontendOptions.initAddrUseIp():269] check ip address: /192.168.198.69
2026-04-14 14:25:42.412+08:00 INFO (main|1) [FrontendOptions.initAddrUseIp():292] Use IP init local addr, IP: /192.168.198.69
2026-04-14 14:25:42.467+08:00 INFO (main|1) [ConsistencyChecker.initWorkTime():133] parsed startDate: 1970-01-01T23:00:00.000+0800, endDate: 1970-01-01T04:00:00.000+0800
2026-04-14 14:25:42.702+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role root[-1]
2026-04-14 14:25:42.706+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role db_admin[-2]
2026-04-14 14:25:42.706+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role cluster_admin[-3]
2026-04-14 14:25:42.707+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role user_admin[-4]
2026-04-14 14:25:42.707+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role security_admin[-106]
2026-04-14 14:25:42.707+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role public[-5]
2026-04-14 14:25:43.097+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [192.168.198.69:9010]
2026-04-14 14:25:43.117+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():340] correct the node name 192.168.124.76_9010_1776088663466 to new node name: 192.168.198.69_9010_1776147943104, role: LEADER
2026-04-14 14:25:43.117+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():508] Current run_mode is shared_data
2026-04-14 14:25:43.117+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():513] Got role: LEADER, node name: 192.168.198.69_9010_1776147943104 and run_mode: shared_data
2026-04-14 14:25:43.121+08:00 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():335] skip check local environment for observer
2026-04-14 14:25:43.121+08:00 ERROR (main|1) [BDBEnvironment.initConfigs():181] Current node is not in the electable_nodes list. will exit
2026-04-14 14:25:43.122+08:00 ERROR (main|1) [StarRocksFE.start():207] StarRocksFE start failed
com.starrocks.journal.JournalException: Current node is not in the electable_nodes list. will exit
        at com.starrocks.journal.bdbje.BDBEnvironment.initConfigs(BDBEnvironment.java:182) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:172) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:151) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1211) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1182) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:143) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:93) ~[starrocks-fe.jar:?]

@Rocky

Rocky

04/14/2026, 6:28 AM

The error "Current node is not in the electable_nodes list" occurs because your FE Pod's IP address changed (from

192.168.124.76

192.168.198.69

). StarRocks stores the IP addresses of nodes allowed to vote in its metadata. Since the new IP isn't in that "electable" list, the node refuses to start to prevent cluster corruption. To fix this and get your cluster running again, follow these steps inside your debug pod: 1. Update
fe.conf
with the recovery flag Open your configuration file:

Copy code

bash
vi /opt/starrocks/fe/conf/fe.conf

Add this line at the end:

Copy code

properties
bdbje_reset_election_group = true

_Note: Ensure

run_mode = shared_data

and your S3/MinIO storage configurations are also in this file, as you are starting it manually._ 2. Start the FE Run the start script:

Copy code

bash
/opt/starrocks/fe/bin/start_fe.sh

3. Verify the FE is the Leader Wait a few seconds, then check the log:

Copy code

bash
tail -f /opt/starrocks/fe/log/fe.log

You should see a message saying the node has become LEADER. Once it is up, connect via MySQL:

Copy code

bash
mysql -h 127.0.0.1 -P9030 -uroot

Run the following command to see the registered frontends:

Copy code

sql
SHOW FRONTENDS\G

You will likely see the old IP listed as

Alive: false

. 4. Important: Clean up Once the FE is running successfully as LEADER: 1. Stop the FE (press

Ctrl+C

or kill the process). 2. Remove the line

bdbje_reset_election_group = true

from

fe.conf

. If you leave this in, it will reset the election group every time the pod restarts, which can cause issues in a multi-FE setup. 3. Restart the FE normally. ────────── Long-term Fix for Kubernetes In Kubernetes, Pod IPs change every time a Pod is recreated. To prevent this error from happening again, you should: 1. Use Fully Qualified Domain Names (FQDN): Add

enable_fqdn_mode = true

to your

fe.conf

. This allows StarRocks to use the K8s Service/Pod DNS names (e.g.,

kube-starrocks-fe-0.starrocks-fe-service

) instead of volatile IPs. 2. Use the StarRocks Operator: The Operator handles these IP changes and configuration injections automatically, preventing the need for manual

bdbje_reset_election_group

Rocky

04/14/2026, 6:28 AM

• administration/management/FE_configuration.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • administration/Meta_recovery.md • deployment/deploy_manually.md • faq/Deploy_faq.md

Abhishek Yadav

04/14/2026, 6:33 AM

still getting 2026-04-14 143209.827+08:00 ERROR (main|1) [BDBEnvironment.initConfigs():181] Current node is not in the electable_nodes list. will exit 2026-04-14 143209.827+08:00 ERROR (main|1) [StarRocksFE.start():207] StarRocksFE start failed com.starrocks.journal.JournalException: Current node is not in the electable_nodes list. will exit @Rocky how to fix it

Rocky

04/14/2026, 6:33 AM

It looks like your FE pod is currently identified as an OBSERVER, which is why the

bdbje_reset_election_group

flag is not working as expected. According to your logs:

skip check local environment for observer

Current node is not in the electable_nodes list. will exit

In StarRocks, only FOLLOWER nodes can be "electable." If the metadata on your disk (from the previous IP) says this node is an Observer, it will refuse to start because it isn't allowed to participate in elections, and it can't find the old Leader to talk to. To fix this and force this node to become the new Leader, follow these exact steps: 1. Change the Node Role to FOLLOWER The FE reads its role from a specific file on disk. You need to manually change it so the "Reset" flag can take effect.

Copy code

bash
# Inside the pod
vi /opt/starrocks/fe/meta/image/ROLE

Change the line

role=OBSERVER

to:

Copy code

properties
role=FOLLOWER

(Keep the other lines like
name
and
hostType
as they are for now; the reset flag will fix them). 2. Ensure
fe.conf
has the Recovery Flag Verify your

/opt/starrocks/fe/conf/fe.conf

contains these lines:

Copy code

properties
run_mode = shared_data
bdbje_reset_election_group = true

(Also ensure your S3/MinIO bucket configurations are present in this file, as you are starting manually and bypassing the entrypoint script). 3. Start the FE Now, run the start script:

Copy code

bash
/opt/starrocks/fe/bin/start_fe.sh

4. Verify and Clean Up Check the log:

tail -f /opt/starrocks/fe/log/fe.log

. You should see: 1. The node successfully resets the BDBJE group. 2. The node promotes itself to LEADER. 3. Once it is running, connect via MySQL (

mysql -P9030

) and run

SHOW FRONTENDS;

. You should see the new IP (

192.168.198.69

) listed as the LEADER. IMPORTANT: Once the FE is running successfully: 1. Stop the FE. 2. Remove

bdbje_reset_election_group = true

from

fe.conf

. 3. Restart the FE normally. Why this is happening in Kubernetes: Your Pod IP changed from

192.168.124.76

192.168.198.69

. Because StarRocks metadata is IP-sensitive, it "locked" itself out. Rocky's Pro-Tip: To prevent this in the future: * Set

enable_fqdn_mode = true

in your

fe.conf

. This makes StarRocks use Kubernetes DNS names (which stay the same) instead of Pod IPs (which change). * Use the StarRocks Operator to manage your deployment; it handles these recovery scenarios automatically. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • administration/management/FE_configuration.md • administration/Meta_recovery.md

Abhishek Yadav

04/14/2026, 6:35 AM

Copy code

2026-04-14 14:34:55.411+08:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [192.168.198.69:9010]
2026-04-14 14:34:55.418+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():508] Current run_mode is shared_data
2026-04-14 14:34:55.419+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():513] Got role: FOLLOWER, node name: 192.168.198.69_9010_1776147943104 and run_mode: shared_data
2026-04-14 14:34:55.421+08:00 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():340] skip check local environment because helper node and local node are identical.
2026-04-14 14:34:55.753+08:00 ERROR (MASTER 192.168.198.69_9010_1776147943104(-1)|1) [StarRocksFE.start():207] StarRocksFE start failed
com.sleepycat.je.DatabaseNotFoundException: (JE 18.3.20) _jeRepGroupDB
        at com.sleepycat.je.rep.impl.RepImpl.openGroupDb(RepImpl.java:1974) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.impl.RepImpl.getGroupDb(RepImpl.java:1912) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.impl.RepGroupDB.reinitFirstNode(RepGroupDB.java:1439) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.impl.node.RepNode.reinitSelfElect(RepNode.java:1688) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.impl.node.RepNode.startup(RepNode.java:875) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.impl.node.RepNode.joinGroup(RepNode.java:2155) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.impl.RepImpl.joinGroup(RepImpl.java:618) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.ReplicatedEnvironment.joinGroup(ReplicatedEnvironment.java:558) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:619) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:464) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:538) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.sleepycat.je.rep.util.DbResetRepGroup.reset(DbResetRepGroup.java:262) ~[starrocks-bdb-je-18.3.20.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initConfigs(BDBEnvironment.java:187) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:172) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:151) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1211) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1182) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:143) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:93) ~[starrocks-fe.jar:?]
root@kube-starrocks-fe-0:/opt/starrocks#

@Rocky getting above issue

Rocky

04/14/2026, 6:37 AM

The new error,

com.sleepycat.je.DatabaseNotFoundException: _jeRepGroupDB

, occurs because the BDBJE Reset Utility (triggered by the

bdbje_reset_election_group

flag) is trying to open the metadata database to fix it, but it cannot find the internal replication group records in your

/opt/starrocks/fe/meta/bdb

directory. This typically happens if the

bdb

directory is empty, corrupted, or does not contain a valid "Replicated Environment" yet. Since you are in Shared-Data mode and trying to recover this node after an IP change, follow these steps to bypass the reset utility and force a metadata recovery: 1. Check your Metadata Directory Ensure you are pointing to the correct data. In your pod, check if there are files in the

bdb

directory:

Copy code

bash
ls -lh /opt/starrocks/fe/meta/bdb

If this directory is empty, the reset flag will always fail because there is no "group" to reset. 2. Switch to Recovery Mode Instead of resetting the election group, we will use StarRocks' built-in Metadata Recovery Mode. This mode allows the FE to start using only the latest "Image" (snapshot) file, ignoring the broken BDB logs. 1. Open

/opt/starrocks/fe/conf/fe.conf

. 2. Remove the line:

bdbje_reset_election_group = true

. 3. Add this line:

Copy code

properties
    metadata_enable_recovery_mode = true

4. Verify

run_mode = shared_data

and your S3/MinIO settings are still in the file. 3. Start the FE Run the start script again:

Copy code

bash
/opt/starrocks/fe/bin/start_fe.sh

4. Verify and Finalize Check the logs:

tail -f /opt/starrocks/fe/log/fe.log

. * The FE should now start up in a "Read-Only" recovery state. * Once it's up, connect via MySQL:

mysql -h 127.0.0.1 -P9030 -uroot

. * Create a new checkpoint image to save the recovered state:

Copy code

sql
    ADMIN SHOW FRONTEND CONFIG LIKE "image_dir"; -- find where images are
    ALTER SYSTEM CREATE IMAGE;

* Wait a minute for the image to be created. 5. Exit Recovery Mode This step is critical. Recovery mode is a temporary state. 1. Stop the FE. 2. Remove

metadata_enable_recovery_mode = true

from

fe.conf

. 3. Restart the FE normally. ────────── Why this is happening (Root Cause) The

_jeRepGroupDB

error specifically means the BDBJE library looks for its system table but finds nothing. This often happens in Kubernetes when: 1. Empty Volume: A new PV was mounted, or the

meta_dir

path in

fe.conf

doesn't match where the actual data is. 2. Incomplete Reset: A previous attempt to clear metadata left the directory in an inconsistent state. Rocky's Pro-Tip: If this node still won't start and your

meta/bdb

is broken, but you have the

meta/image

files, you can delete everything inside the
bdb
folder and then use the

metadata_enable_recovery_mode = true

flag. This forces the FE to reconstruct the BDB environment from the snapshot image. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • faq/Deploy_faq.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md

Rocky

04/14/2026, 6:37 AM

Rocky

04/14/2026, 6:37 AM

Rocky

04/14/2026, 6:37 AM

Rocky

04/14/2026, 6:37 AM

Rocky

04/14/2026, 6:37 AM

Open in Slack

Previous Next