< Rocky> I am seeing this error in my starrocks logs and all StarRocks #questions-and-troubleshooting

<@U0A71G31CDV>, I am seeing this error in my starr...

kushagra.dubey

08/20/2026, 7:05 AM

@Rocky, I am seeing this error in my starrocks logs and all my fe pods are down- OWER ==> /opt/starrocks/fe/log/fe.out <== [2026-08-20 063022] notify new FE type transfer: FOLLOWER [Thu Aug 20 063023 UTC 2026] Waiting to register as follower, retrying... [Thu Aug 20 063028 UTC 2026] Waiting to register as follower, retrying... ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 063032.442Z WARN (UNKNOWN starrocks-fe-2.starrocks-fe-search.trino.svc.cluster.local_9010_1778607365426(-1)|1) [ColocateTableIndex.cleanupInvalidDbOrTable():940] remove 0 invalid tableid: [] 2026-08-20 063032.693Z WARN (UNKNOWN starrocks-fe-2.starrocks-fe-search.trino.svc.cluster.local_9010_1778607365426(-1)|1) [GlobalStateMgr.loadImage():1699] Miss meta block [20001], This may not be a fatal error. It may be because there are new features in the version you upgraded this time, but there is no relevant metadata. [Thu Aug 20 063033 UTC 2026] Waiting to register as follower, retrying... [Thu Aug 20 063038 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063042.202Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20143584 2026-08-20 063042.202Z WARN (replayer|134) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787207442202, synchronized time: 1787175366668, has log: true, fe type: FOLLOWER 2026-08-20 063043.206Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20143999 [Thu Aug 20 063043 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063044.211Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20144350 2026-08-20 063045.214Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20144732 2026-08-20 063046.258Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1042 replayedJournalId: 20145200 2026-08-20 063047.262Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20145531 2026-08-20 063047.264Z WARN (replayer|134) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787207447264, synchronized time: 1787175386680, has log: true, fe type: FOLLOWER 2026-08-20 063048.269Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 [Thu Aug 20 063114 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063114.826Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20173278 2026-08-20 063115.830Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20173389 2026-08-20 063116.835Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20173852 2026-08-20 063117.839Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20174320 2026-08-20 063117.840Z WARN (replayer|134) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787207477840, synchronized time: 1787176087836, has log: true, fe type: FOLLOWER 2026-08-20 063118.843Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20174680 [Thu Aug 20 063119 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063119.849Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1004 replayedJournalId: 20175310 2026-08-20 063120.855Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20175640 2026-08-20 063121.860Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1004 replayedJournalId: 20176061 2026-08-20 063123.284Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1421 replayedJournalId: 20176258 2026-08-20 063123.284Z WARN (replayer|134) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787207483284, synchronized time: 1787176097841, has log: true, fe type: FOLLOWER [Thu Aug 20 063124 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063124.289Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20176724 2026-08-20 063125.291Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20177173 2026-08-20 063126.295Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20177586 2026-08-20 063127.302Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1005 replayedJournalId: 20178008 2026-08-20 063128.308Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1005 replayedJournalId: 20178445 2026-08-20 063128.309Z WARN (replayer|134) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787207488309, synchronized time: 1787176117848, has log: true, fe type: FOLLOWER [Thu Aug 20 063129 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063129.313Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20178847 2026-08-20 063130.316Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20180080 2026-08-20 063131.320Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20180746 2026-08-20 063132.322Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20181366 2026-08-20 063133.325Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20191333 2026-08-20 063133.327Z WARN (replayer|134) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787207493326, synchronized time: 1787176524936, has log: true, fe type: FOLLOWER [Thu Aug 20 063134 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063134.331Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20191780 2026-08-20 063135.334Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20192229 2026-08-20 063136.341Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1005 replayedJournalId: 20192799 2026-08-20 063137.343Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20193349 2026-08-20 063138.346Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20193873 2026-08-20 063138.348Z WARN (replayer|134) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787207498348, synchronized time: 1787176544947, has log: true, fe type: FOLLOWER [Thu Aug 20 063139 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 063139.352Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20194343 2026-08-20 063140.355Z WARN (replayer|134) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20195171 2026-08-20 063141.200Z WARN (UNKNOWN starrocks-fe-2.starrocks-fe-search.trino.svc.cluster.local_9010_1778607365426(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. ]

Rocky

08/20/2026, 7:06 AM

The errors in your logs indicate that your StarRocks FE (Front End) nodes are stuck in a replaying state and are unable to elect a leader or join the cluster. Analysis of the Logs 1. Meta Out of Date / Massive Lag: The log

meta out of date. current time: ..., synchronized time: ...

shows a synchronized time that is roughly 9 hours behind the current time. This means the FE is replaying a massive amount of historical metadata journals. 2. Slow Journal Replay: The warning

replay journal cost too much time: 1001 replayedJournalId: 20143584

indicates that the "replayer" thread is processing journals very slowly (about 1 second per batch). Since the

replayedJournalId

is increasing in your logs, the FE is actually progressing, but it has millions of entries to catch up on. 3. Checkpoint Failure: The most likely cause is that your FE failed to create a "checkpoint" (an image file) for a long time. When an FE restarts, it must replay all journals since the last image file. If no image was created for 9 hours, the startup will take hours. 4. Waiting to Register: The message

Waiting to register as follower

usually comes from a startup wrapper script (common in K8s/Docker) that waits for the FE to reach a stable state. Because the FE is busy replaying logs, it hasn't reached the "Ready" state yet. Recommended Steps 1. Monitor Progress Check if the

replayedJournalId

is still increasing in your

fe.log

. * If it is increasing: The FE is not "dead"; it is just busy. You can estimate how long it will take by seeing how many journals are left. * If it is stuck: The replayer thread might be blocked (possibly by Disk I/O or GC). 2. Check the Metadata Directory Check the

meta/image

and

meta/bdb

directories on your FE pods: * Look at the latest file in

meta/image/

. If the timestamp is very old, it confirms the checkpointing failure. * Check the size of

meta/bdb/

. If it is many gigabytes, it confirms there are too many logs to replay. 3. Address Resource Constraints Journal replay is CPU and Memory intensive. Check your K8s pod resources: * Increase JVM Heap: Ensure your FE has enough memory (at least 8GB-16GB recommended). Checkpoint failures are often caused by Full GC preventing the FE from writing the image file. * Check Disk I/O: Slow journal replay is often caused by slow disk performance on the metadata volume. 4. Force a Leader (If the cluster is stuck) If all FE pods are stuck as

FOLLOWER

and none become

LEADER

after a reasonable wait, the election group might be corrupted or unable to reach a quorum. 1. Identify the pod with the most recent metadata (the one with the highest

replayedJournalId

). 2. Stop all FE pods. 3. Add the following line to the

fe.conf

of that one pod:

Copy code

properties
    bdbje_reset_election_group = true

4. Restart that pod. It should force itself to become the Leader. 5. Important: Once it becomes Leader, remove that line from

fe.conf

and restart it again, then bring up the other FEs as followers. 5. "Miss meta block [20001]" This specific warning is common after an upgrade to StarRocks 3.x. It indicates the FE is looking for new metadata structures that don't exist yet in your old image. This is generally non-fatal and can be ignored once the cluster is stable. Summary Your FEs are likely not crashed, but are "frozen" while catching up on 9 hours of missed metadata. If the

replayedJournalId

is moving, the safest path is to wait. If you cannot wait, you may need to perform a Metadata Recovery by resetting the election group on the node with the latest data. References • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 7:18 AM

@Rocky, for FE-0 I can see these logs - 2026-08-20 071304.261Z WARN (replayer|119) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787209984261, synchronized time: 1787176067828, has log: true, fe type: FOLLOWER 2026-08-20 071305.264Z WARN (replayer|119) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20172286 2026-08-20 071306.778Z WARN (replayer|119) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1512 replayedJournalId: 20172478 2026-08-20 071307.781Z WARN (replayer|119) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20172864 2026-08-20 071308.785Z WARN (replayer|119) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20173145 For Fe1- 2026-08-20 071759.842Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20171003 [Thu Aug 20 071759 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 071800.844Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20171387 2026-08-20 071801.846Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20171806 2026-08-20 071802.852Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20172251 2026-08-20 071802.852Z WARN (replayer|144) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787210282852, synchronized time: 1787176067828, has log: true, fe type: UNKNOWN 2026-08-20 071803.870Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1016 replayedJournalId: 20172629 2026-08-20 071804.873Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20172722 [Thu Aug 20 071804 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 071805.880Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1005 replayedJournalId: 20173139 2026-08-20 071806.883Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20173591 2026-08-20 071807.888Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20174164 2026-08-20 071807.889Z WARN (replayer|144) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787210287889, synchronized time: 1787176087836, has log: true, fe type: UNKNOWN 2026-08-20 071808.892Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20174707 2026-08-20 071809.896Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20175445 [Thu Aug 20 071809 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 071811.163Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1265 replayedJournalId: 20175685 2026-08-20 071812.168Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20176145 2026-08-20 071813.172Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20176591 2026-08-20 071813.173Z WARN (replayer|144) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787210293172, synchronized time: 1787176097841, has log: true, fe type: UNKNOWN 2026-08-20 071814.179Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1005 replayedJournalId: 20176894 [Thu Aug 20 071814 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 071815.182Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20177305 2026-08-20 071815.535Z ERROR (star mgr replayer|143) [Daemon.run():110] daemon thread exits. name=star mgr replayer 2026-08-20 071816.188Z WARN (replayer|144) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1004 replayedJournalId: 20177688 2026-08-20 071816.190Z ERROR (replayer|144) [Daemon.run():110] daemon thread exits. name=replayer [Thu Aug 20 071819 UTC 2026] Waiting to register as follower, retrying... [Thu Aug 20 071824 UTC 2026] Waiting to register as follower, retrying... For Fe-2 2026-08-20 071840.835Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20176616 2026-08-20 071841.837Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20176956 2026-08-20 071842.841Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20177348 2026-08-20 071843.846Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20177756 2026-08-20 071844.848Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20178132 2026-08-20 071844.849Z WARN (replayer|138) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787210324849, synchronized time: 1787176107844, has log: true, fe type: FOLLOWER [Thu Aug 20 071845 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 071845.852Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20178508 2026-08-20 071846.855Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20179563 2026-08-20 071847.859Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20180228 2026-08-20 071848.237Z WARN (UNKNOWN starrocks-fe-2.starrocks-fe-search.trino.svc.cluster.local_9010_1778607365426(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 071848.862Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20180737 2026-08-20 071849.865Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1002 replayedJournalId: 20181289 2026-08-20 071849.866Z WARN (replayer|138) [GlobalStateMgr$5.setCanRead():2014] meta out of date. current time: 1787210329866, synchronized time: 1787176137855, has log: true, fe type: FOLLOWER [Thu Aug 20 071850 UTC 2026] Waiting to register as follower, retrying... 2026-08-20 071850.869Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20184945 2026-08-20 071851.871Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1001 replayedJournalId: 20191516 2026-08-20 071852.875Z WARN (replayer|138) [GlobalStateMgr.replayJournalInner():2146] replay journal cost too much time: 1003 replayedJournalId: 20191867

Rocky

08/20/2026, 7:19 AM

The logs you provided show that your StarRocks FE cluster is stuck replaying a massive backlog of metadata (journals) and is currently unable to elect a leader. Key Observations from your Logs: 1. Massive Metadata Lag: All FE nodes are replaying journals from about 9.5 hours ago (

synchronized time: 1787176067828

). 2. Slow Progress: The

replay journal cost too much time: 1001

warnings show that the replayer is processing journals slowly. At roughly 1,000 IDs per second, replaying millions of entries will take hours. 3. FE-1 has Failed: Crucially, on FE-1, the replayer thread has crashed:

ERROR (replayer|144) [Daemon.run():110] daemon thread exits. name=replayer

. This node is now unhealthy and will not catch up or participate in elections until restarted and the root cause (often OutOfMemory or disk issues) is fixed. 4. No Leader: Because FE-1 is down and FE-0/FE-2 are still replaying old data, the cluster cannot reach a "Ready" state or form a quorum. 5. Restart Loop: The

Waiting to register as follower, retrying...

message suggests a startup script is waiting for the FE to become healthy. If this script has a timeout, it might be restarting the FE process repeatedly, preventing it from ever finishing the 9-hour replay. Recommended Recovery Steps 1. Check for Resource Issues (FE-1) Check the

fe.log

on FE-1 specifically for the lines immediately before the

daemon thread exits

error. Look for a Java Stack Trace or

java.lang.OutOfMemoryError

. * Action: If it's an OOM, you must increase the JVM heap size (

-Xmx

fe.conf

) for all FE nodes to handle the large metadata backlog. 2. Identify the Node with the Latest Data Compare the

replayedJournalId

in the latest logs of all pods: * FE-0: ~20,173,145 * FE-1: ~20,177,688 (Crashed) * FE-2: ~20,191,867 (Most advanced) FE-2 has the most recent metadata. 3. Force a Leader Election (Manual Recovery) Since the cluster is stuck and one node's replayer has exited, you likely need to force one node to become the leader to break the cycle. 1. Scale down or stop all FE pods/processes. 2. On the node with the most recent data (FE-2), add this line to your `fe.conf`:

Copy code

properties
    metadata_failure_recovery = true

_(Note: In older versions, this might be

bdbje_reset_election_group = true

. Check your version, but

metadata_failure_recovery

is the modern standard for forcing a leader.)_ 3. Start FE-2 only. It should ignore the other nodes and force itself to become the LEADER. 4. Check the

fe.log

of FE-2. Once it says

transfer to LEADER

, immediately remove the

metadata_failure_recovery

line from

fe.conf

. 5. Restart FE-2 again (without the recovery flag). 6. Start FE-0 and FE-1. They should now join FE-2 as followers and sync metadata from it. Warning on the "Waiting to register" Loop If you are running on Kubernetes, ensure your Liveness/Readiness probes are not killing the pods while they are replaying journals. Replaying 9 hours of logs is a heavy operation; if the pod is killed and restarted halfway through, it will start from the last checkpoint again, leading to an infinite loop. You may need to temporarily increase the

initialDelaySeconds

failureThreshold

of your probes. References • administration/Meta_recovery.md • faq/loading_faq.md (General FE Troubleshooting)

kushagra.dubey

08/20/2026, 7:31 AM

@Rocky, these are the logs after scalling it down to 1 pod - start time: Thu Aug 20 072823 UTC 2026, server uptime: 072823 up 20 days, 23:43, 0 users, load average: 4.46, 3.58, 3.22 ===== <?xml version="1.0" encoding="utf-8"?> <Configuration status="info" packages="com.starrocks.common"> <Appenders> <Console name="ConsoleErr" target="SYSTEM_ERR" follow="true"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} %p (%t|%tid) [%C{1}.%M():%L] %m%n"/> </Console> <RollingFile name="Sys" fileName="/opt/starrocks/fe/log/fe.log" filePattern="/opt/starrocks/fe/log/fe.log.%d{yyyyMMdd}-%i.gz"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} %p (%t|%tid) [%C{1}.%M():%L] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="10" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.log.*" /> <IfLastModified age="1d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="SysWF" fileName="/opt/starrocks/fe/log/fe.warn.log" filePattern="/opt/starrocks/fe/log/fe.warn.log.%d{yyyyMMdd}-%i.gz"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} %p (%t|%tid) [%C{1}.%M():%L] %m%n %ex"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="10" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.warn.log.*" /> <IfLastModified age="1d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="Auditfile" fileName="/opt/starrocks/fe/log/fe.audit.log" filePattern="/opt/starrocks/fe/log/fe.audit.log.%d{yyyyMMdd}-%i"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} [%c{1}] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="5" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.audit.log.*" /> <IfLastModified age="30d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="dumpFile" fileName="/opt/starrocks/fe/log/fe.dump.log" filePattern="/opt/starrocks/fe/log/fe.dump.log.%d{yyyyMMdd}-%i"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} [%c{1}] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="10" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.dump.log.*" /> <IfLastModified age="1d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="BigQueryFile" fileName="/opt/starrocks/fe/log/fe.big_query.log" filePattern="/opt/starrocks/fe/log/fe.big_query.log.%d{yyyyMMdd}-%i"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} [%c{1}] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="10" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.big_query.log.*" /> <IfLastModified age="7d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="ProfileFile" fileName="/opt/starrocks/fe/log/fe.profile.log" filePattern="/opt/starrocks/fe/log/fe.profile.log.%d{yyyyMMdd}-%i"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} [%c{1}] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="5" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.profile.log.*" /> <IfLastModified age="1d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="FeaturesFile" fileName="/opt/starrocks/fe/log/fe.features.log" filePattern="/opt/starrocks/fe/log/fe.features.log.%d{yyyyMMdd}-%i"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} [%c{1}] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="5" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.features.log.*" /> <IfLastModified age="1d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="InternalFile" fileName="/opt/starrocks/fe/log/fe.internal.log" filePattern="/opt/starrocks/fe/log/fe.internal.log.%d{yyyyMMdd}-%i"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} %p (%t|%tid) [%C{1}.%M():%L] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="90" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.internal.log.*" /> <IfLastModified age="7d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> <RollingFile name="PlanFile" fileName="/opt/starrocks/fe/log/fe.plan.log" filePattern="/opt/starrocks/fe/log/fe.plan.log.%d{yyyyMMdd}-%i"> <PatternLayout charset="UTF-8" pattern="%d{yyyy-MM-dd HHmmss.SSSXXX} [%c{1}] %m%n"/> <Policies> <TimeBasedTriggeringPolicy/> <SizeBasedTriggeringPolicy size="1024MB"/> </Policies> <DefaultRolloverStrategy max="10" fileIndex="min"> <Delete basePath="/opt/starrocks/fe/log/" maxDepth="1" followLinks="true"> <IfFileName glob="fe.plan.log.*" /> <IfLastModified age="7d" /> </Delete> </DefaultRolloverStrategy> </RollingFile> </Appenders> <Loggers> <Root level="WARN"> <AppenderRef ref="Sys"/> <AppenderRef ref="SysWF" level="WARN"/> </Root> <Logger name="audit" level="ERROR" additivity="false"> <AppenderRef ref="Auditfile"/> </Logger> <Logger name="dump" level="ERROR" additivity="false"> <AppenderRef ref="dumpFile"/> </Logger> <Logger name="big_query" level="ERROR" additivity="false"> <AppenderRef ref="BigQueryFile"/> </Logger> <Logger name="profile" level="INFO" additivity="false"> <AppenderRef ref="ProfileFile"/> </Logger> <Logger name="features" level="INFO" additivity="false"> <AppenderRef ref="FeaturesFile"/> </Logger> <Logger name="plan" level="INFO" additivity="false"> <AppenderRef ref="PlanFile"/> </Logger> <Logger name='internal.base' level="INFO" additivity="false"> <AppenderRef ref="InternalFile"/> </Logger> <Logger name='internal.statistic' level="INFO" additivity="false"> <AppenderRef ref="InternalFile"/> </Logger> <Logger name='audit.slow_query' level='INFO'/> <Logger name='dump.query' level='INFO'/> <Logger name='big_query.query' level='INFO'/> <Logger name='org.apache.kafka' level='WARN'><AppenderRef ref='SysWF'/></Logger> <Logger name='org.apache.hudi' level='WARN'><AppenderRef ref='SysWF'/></Logger> <Logger name='org.apache.hadoop.io.compress' level='WARN'><AppenderRef ref='SysWF'/></Logger> </Loggers> </Configuration> ===== ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 072826.381Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN ==> /opt/starrocks/fe/log/fe.out <== [2026-08-20 072826] notify new FE type transfer: UNKNOWN ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 072845.305Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [ColocateTableIndex.cleanupInvalidDbOrTable():940] remove 0 invalid tableid: [] 2026-08-20 072845.583Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.loadImage():1699] Miss meta block [20001], This may not be a fatal error. It may be because there are new features in the version you upgraded this time, but there is no relevant metadata. 2026-08-20 072954.217Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 073054.236Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details.

Rocky

08/20/2026, 7:32 AM

The reason your FE is currently stuck in the

UNKNOWN

state after scaling down to one pod is that StarRocks requires a quorum (a majority of nodes) to elect a leader. In a cluster originally configured with 3 FE nodes, a single node cannot form a quorum by itself and will wait indefinitely for others to join. To force this single node to become the leader and start the cluster, follow these recovery steps: Phase 1: Force Leader Recovery 1. Stop the FE-0 pod. 2. Edit the
fe.conf
file for FE-0 and add the following line:

Copy code

properties
    metadata_failure_recovery = true

3. Start the FE-0 pod. 4. Monitor the
fe.log
. You are looking for a log entry that says:

notify new FE type transfer: LEADER

Note: It will still need to replay those 9 hours of journals you saw earlier. This might still take some time, but it should now progress. Phase 2: Cleanup and Cluster Expansion Once the FE-0 pod successfully transfers to

LEADER

and is able to accept SQL queries (check via

mysql

client): 1. Remove the flag: Edit

fe.conf

again and remove the line

metadata_failure_recovery = true

. 2. Restart FE-0: Restart the pod one more time without the flag. This is a critical step to prevent metadata issues later. 3. Check Image Creation: In the MySQL client, run:

Copy code

sql
    SHOW FRONTENDS;

Ensure the node is

IsMaster: true

. Then, force a new checkpoint to shrink the journal backlog:

Copy code

sql
    ALTER SYSTEM CREATE IMAGE;

4. Re-scale the cluster: You can now scale your FE pods back up to 3. The other nodes will join as followers and sync the fresh image from FE-0. Important Troubleshooting Notes * Replayer Crashes: In your previous logs, FE-1's

replayer

thread crashed. This is almost always caused by an OutOfMemoryError. Ensure your FE pods have at least 8GB-16GB of JVM Heap (configured via

-Xmx

fe.conf

) to handle the large metadata replay. * Probe Timeouts: If you are on Kubernetes, the pod might restart itself if the

Liveness

Readiness

probes fail while it's busy replaying logs. Temporarily increase the

initialDelaySeconds

failureThreshold

of your probes to give the FE time to finish its startup. References • administration/Meta_recovery.md References • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 7:35 AM

@Rocky Currently my starrocks version is 4.0.9, will these recovery steps, are these recovery steps applicable to this version

Rocky

08/20/2026, 7:35 AM

Yes, the recovery steps are applicable to version 4.0.9, but there is one important change to the configuration parameter based on your version. In StarRocks v3.3 and later (including your version 4.0.9), the older parameter

metadata_failure_recovery

has been replaced by a more specific one for resetting the BDBJE election group. Updated Recovery Steps for v4.0.9: 1. Stop the FE-0 pod. 2. Edit
fe.conf
and use this specific parameter instead:

Copy code

properties
    bdbje_reset_election_group = true

_(Note: Using the old

metadata_failure_recovery = true

may not work or is deprecated in 4.x versions.)_ 3. Start the FE-0 pod. * Since you have scaled down to 1 pod, this flag will force this single pod to ignore the "missing" majority and elect itself as the LEADER. * Monitor

fe.log

. It will likely still spend a significant amount of time replaying those 9+ hours of journals. Do not interrupt it while it is replaying, or you may risk metadata corruption. 4. Verify Leader Status: Once the logs show

transfer to LEADER

, log in via a MySQL client and run:

Copy code

sql
    SHOW FRONTENDS;

Ensure the node is

Alive: true

and

Role: LEADER

. 5. CRITICAL: Remove the flag. After the FE is successfully running as Leader, remove

bdbje_reset_election_group = true

from

fe.conf

and restart the FE pod one more time. If you leave this flag in, the FE will reset its election group every time it restarts, which will prevent other FEs from joining correctly later. 6. Scale back up: Once FE-0 is stable and the flag is removed, you can scale your FE deployment back to 3 pods. The new pods will automatically join as followers. Why this is happening in 4.0.9 Because your cluster was originally a 3-node HA cluster, StarRocks requires 2 nodes to be healthy to perform any operations. By scaling down to 1 pod, you took away the "quorum." The

bdbje_reset_election_group

flag tells the remaining node to "forget" the old 3-node group and start a new group where it is the sole member/leader. References • administration/Meta_recovery.md (See the version-specific notes for v3.3+)

kushagra.dubey

08/20/2026, 7:43 AM

@Rocky These are my starrocks config helm charts, can I add this value here -

Copy code

config: |
        LOG_DIR = /opt/starrocks/fe/log
        DATE_FORMAT = %Y-%m-%d %H:%M:%S
        # Shared-data mode: store all data in GCS
        run_mode = shared_data
        cloud_native_meta_port = 6090
        enable_load_volume_from_conf = true
        cloud_native_storage_type = S3
        max_query_num = 1000
        query_timeout = 300
        query_port = 9030
        sys_log_level = WARN
        sys_log_roll_interval = DAY
        sys_log_roll_num = 10
        sys_log_delete_age = 1d
        sys_log_roll_mode = SIZE-MB-512
        sys_log_enable_compress = true
        audit_log_modules = slow_query
        audit_log_roll_num = 5
        audit_log_roll_interval = DAY
        audit_log_delete_age = 30d
        dump_log_delete_age = 1d
        feature_log_delete_age = 1d
        enable_profile = false
        query_detail_size = 100
        enable_collect_full_statistic = false
        enable_statistic_collect = false
        enable_statistic_collect_on_first_load = false
        iceberg_worker_num_threads = 8
        lake_batch_publish_max_version_num = 16
        max_running_txn_num_per_db = 1000
        publish_version_max_threads = 64
        query_deploy_threadpool_size = 80

Rocky

08/20/2026, 7:44 AM

Yes, you can add it directly to that

config

block in your Helm chart. How to apply it via Helm: 1. Add the flag: Update your

values.yaml

(or the specific configuration section in your chart) to include the line:

Copy code

yaml
    config: |
      bdbje_reset_election_group = true
      LOG_DIR = /opt/starrocks/fe/log
      # ... rest of your config

2. Apply the change: Run

helm upgrade

to update the ConfigMap and restart the FE pod. 3. Monitor the logs: * Use

kubectl logs -f <fe-pod-name>

to watch the replay process. * Wait until you see:

notify new FE type transfer: LEADER

. 4. CRITICAL - Remove the flag immediately: Once the FE is a

LEADER

, you must remove that line from your Helm chart and run

helm upgrade

again. * Why? If you leave it in, every time the FE pod restarts (due to node maintenance or crashes), it will wipe its knowledge of the other FE nodes, preventing you from ever scaling back up to a 3-node High Availability cluster. Important note for
shared_data
mode: Since you are running in shared_data mode (Cloud-Native), your table data is in GCS, but your metadata (the journal/edit logs) is still stored on the local Persistent Volume (PV) assigned to the FE pod. This recovery flag only affects those local metadata logs. Summary of the full workflow: 1. Scale to 1 FE replica (already done). 2. Add

bdbje_reset_election_group = true

to Helm config and

helm upgrade

. 3. Wait for the FE to finish replaying logs and become

LEADER

. 4. Remove the flag from Helm config and

helm upgrade

. 5. Scale back to 3 FE replicas. One final check: Ensure your FE pod's Liveness/Readiness probes in the Helm chart are not too aggressive. Because you have a large metadata backlog (9+ hours), the FE might take 10-30 minutes to finish replaying. If K8s kills the pod because it's not "Ready" yet, it will never finish the recovery. You may want to temporarily increase the

initialDelaySeconds

for your probes.

kushagra.dubey

08/20/2026, 7:54 AM

@Rocky, these are the logs after making replicas as 1 with the bdbje_reset_election_group = true property - ===== ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 075203.421Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN ==> /opt/starrocks/fe/log/fe.out <== [2026-08-20 075203] notify new FE type transfer: UNKNOWN ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 075203.429Z WARN (RepNode starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|104) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: LEADER ==> /opt/starrocks/fe/log/fe.out <== [2026-08-20 075203] notify new FE type transfer: LEADER ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 075213.323Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [ColocateTableIndex.cleanupInvalidDbOrTable():940] remove 0 invalid tableid: [] 2026-08-20 075213.597Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.loadImage():1699] Miss meta block [20001], This may not be a fatal error. It may be because there are new features in the version you upgraded this time, but there is no relevant metadata. 2026-08-20 075321.933Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2

Rocky

08/20/2026, 7:55 AM

It looks like the actual log output was missing from your message. However, based on the setting you applied, here is exactly what you should look for in your

fe.log

to know if it is working: 1. The Success Marker You are waiting for this specific line:

Copy code

text
notify new FE type transfer: LEADER

Once you see this, the recovery is successful. The FE is now the master of the cluster and you can connect to it via MySQL. 2. The Progress Marker (Journal Replay) Since your previous logs showed a 9-hour gap, the FE must replay all those logs before it can become the Leader. Look for lines like:

Copy code

text
replayed journal id 123456...

If you see these numbers increasing, the FE is working through the backlog. Do not restart the pod while this is happening, as interrupting a massive replay can sometimes lead to metadata inconsistencies. 3. Potential Errors to Watch For If it is still stuck in

UNKNOWN

or crashing, please check for these common errors: * EnvironmentFailureException
: This usually means the BDBJE metadata on disk is corrupted. * InsufficientLogException
: This happens if the journals needed for recovery were already cleaned up. * IP/FQDN Mismatch: If you are not using FQDNs and the pod's IP changed during the scale-down, BDBJE might refuse to start because it doesn't recognize itself in the metadata. Please paste the last 50-100 lines of your
fe.log
if it is still not reaching the
LEADER
state. Reminder: The "Safety" Step As soon as you see

transfer to LEADER

, you must: 1. Remove

bdbje_reset_election_group = true

from your Helm chart. 2. Run
helm upgrade
to restart the pod without the flag. Failure to do this will cause the FE to reset its group again on the next restart, which will prevent you from scaling back up to 3 replicas.

Rocky

08/20/2026, 7:55 AM

The logs show that the recovery flag is working exactly as expected. Here is what is happening: 1. BDBJE Success: You see the log

notify new FE type transfer: LEADER

. This means the node has successfully forced itself to become the master of the cluster. 2. Metadata Replay in Progress: The

GlobalStateMgr.waitForReady

warning is appearing because the FE is currently replaying those 9 hours of journal logs you had earlier. Replaying a large volume of metadata takes time, and while it is replaying, the FE remains in an "initializing" state. 3. The "UNKNOWN" Thread Name: Even though the underlying election layer is a

LEADER

, the StarRocks application thread still labels itself as

UNKNOWN

until the replay is finished and it is 100% ready to serve traffic. What to do now: * Wait: Do not restart the pod or change any settings. Replaying 9 hours of journals can take anywhere from 10 to 30 minutes (or longer depending on the number of transactions and disk speed). * Watch for Progress: Keep tailing the

fe.log

. Look for lines that mention

replayed journal id: XXXXX

. If that number is increasing, the FE is healthy and working through the backlog. * Check the Query Port: Once the replay is finished, you will see a log entry like

Finished to replay journal

and the FE will start listening on port

. When to worry: If the logs stop moving entirely for more than 15 minutes or you see an

OutOfMemoryError

, then the replayer is stuck. Given your 9-hour backlog, the most common issue is that the FE needs more memory. If it crashes, increase the JVM Heap size (

-Xmx

) in your Helm chart and try again. Summary: You have successfully cleared the "no quorum" hurdle. Now you just need to let the FE finish its homework (replaying the logs) before it can fully start up. Next Step (Once it's running): As soon as you can connect via MySQL, remember to remove the

bdbje_reset_election_group

flag from your Helm chart and update, otherwise you won't be able to add the other 2 replicas back safely. References • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 7:55 AM

Rocky

08/20/2026, 7:55 AM

These logs indicate that the recovery is working, but your FE is currently in a long-running replay phase. Breakdown of the logs: 1. notify new FE type transfer: LEADER
: This is a success. It means the

bdbje_reset_election_group

flag worked, and this single FE has successfully elected itself as the Leader of a new 1-node group. 2. Miss meta block [20001]
: This is a common informational warning in StarRocks 4.0. It usually occurs because your metadata image doesn't contain certain new feature blocks (like StarManager or RBAC updates) that the new version expects. As the log says, "This may not be a fatal error." It will continue past this. 3. GlobalStateMgr.waitForReady() ... It took too much time
: This is the most important part. It is a timeout warning telling you that the FE has been trying to start for several minutes but isn't "Ready" to serve queries yet. * In your case, this is expected because you have 9 hours of journal logs to replay. * The FE must finish replaying every single one of those journals before it transitions from

UNKNOWN

LEADER

at the application level. Critical Next Steps: 1. Do NOT restart the pod: The replay process is happening in the background. If you restart it, it has to start replaying from the last image file all over again. 2. Check for Progress: Run the following command inside the FE pod (or tail the log) to see if it is actually replaying:

Copy code

bash
    grep "replayed journal id" /opt/starrocks/fe/log/fe.log | tail -n 5

If the

journal id

numbers are increasing, the recovery is working perfectly. You just need to wait. 3. Verify Kubernetes Probes: Check

kubectl describe pod <fe-pod-name>

. If you see "Liveness probe failed" or "Restarts: 1", Kubernetes is killing the FE because it thinks it’s stuck. * Action: If it's restarting, you must temporarily increase the

initialDelaySeconds

failureThreshold

of your FE Liveness and Readiness probes in your Helm chart to give it 30–60 minutes to finish the replay. 4. Monitor Resources: Ensure the FE pod has enough CPU and Memory. Replaying 9 hours of logs is CPU-intensive. If the pod hits an OOM (Out of Memory), you will need to increase the FE JVM heap size. Summary: Your FE has successfully cleared the "no quorum" state. It is now simply "doing its homework" by replaying the 9-hour backlog. Once it finishes, the

UNKNOWN

in your logs will change to

LEADER

, and port

will open. References • administration/Meta_recovery.md (See StarRocks metadata damage section) References • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 7:58 AM

@Rocky, I don't see any logs after this - ==> /opt/starrocks/fe/log/fe.out <== [2026-08-20 075703] notify new FE type transfer: LEADER ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 075712.961Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [ColocateTableIndex.cleanupInvalidDbOrTable():940] remove 0 invalid tableid: [] 2026-08-20 075713.229Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.loadImage():1699] Miss meta block [20001], This may not be a fatal error. It may be because there are new features in the version you upgraded this time, but there is no relevant metadata.

Rocky

08/20/2026, 7:58 AM

This is actually normal and expected. You are in the middle of the "Loading Image" phase, which is the most silent part of the StarRocks startup sequence. 1. Why the logs stopped When you see the message

Miss meta block [20001]

, the FE has just started reading the

image.XXXX

file from your disk. * The Warning is Normal: The

Miss meta block [20001]

warning is standard in newer versions (like 3.3+ or 4.0). It just means your current metadata image doesn't have specific "Cloud Native" or "RBAC" blocks that the new version expects. It is not an error and will not stop the process. * Silent Period: During the time the FE is loading the image into memory, it does not log every single row it reads. If your metadata image is large (hundreds of MBs or GBs), this phase can take 5 to 10 minutes with no log output at all. 2. How to tell if it's actually working Since you've only waited about a minute (based on the timestamps), it's likely just busy. You can verify it's alive by running this from your terminal: * Check CPU Usage:

Copy code

bash
    kubectl top pod <fe-pod-name>

If the FE pod is using significant CPU (e.g., 100% of a core or more), it is actively loading the metadata into memory. * Check Memory Usage: Ensure the memory isn't flatlining at its limit. If it hits the limit, it might be stuck in a "GC loop" (Garbage Collection). * Check for Crashes:

Copy code

bash
    kubectl get pod <fe-pod-name>

RESTARTS

is 0 and the state is

Running

, it’s still working. 3. What comes next? Once the image is loaded, you will suddenly see a burst of logs like this: 1.

Finished to load image in XXX ms

start to replay journal from id XXX to YYY

3. The Journal Replay: This is where it will process those 9 hours of missing data. You will see lines like

replayed journal id: 123456

. Summary Recommendation: Wait at least 15 minutes. With a 9-hour backlog and a large image, the FE is doing a lot of heavy lifting in the background. As long as the pod doesn't restart, it is making progress. One Warning: If the pod does restart, check

kubectl logs -p <pod-name>

(previous logs). If you see

java.lang.OutOfMemoryError

, you will need to increase the FE memory in your Helm chart before it can successfully load this much metadata. References • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 8:00 AM

@Rocky, now I can see restarts happenign - starrocks-fe-0 0/1 Running 1 (3m32s ago) 8m42s

Rocky

08/20/2026, 8:01 AM

The fact that your pod is restarting is likely due to one of two "protective" mechanisms in Kubernetes or the FE itself. Because the FE is busy replaying 9 hours of journals, it isn't responding to health checks. Step 1: Identify the "Killer" Run this command immediately to see why the previous container died:

Copy code

bash
kubectl describe pod starrocks-fe-0

Look for the Last State: Terminated
section: * If Reason is `OOMKilled`: Your FE doesn't have enough RAM to load the metadata image and the 9-hour journal backlog. * Fix: Increase the Memory Request/Limit in your Helm chart and increase the

-Xmx

value in

FE_JVM_OPTS

. * If Reason is
Completed
or
Error
(and you see Liveness probe failures in Events): Kubernetes killed the FE because it took too long to start. * Fix: You must temporarily disable or significantly extend your probes (see Step 2). Step 2: Fix Probe Timeouts (The most likely cause) During a metadata recovery, the FE is "live" but not "ready." Kubernetes will kill it if the

livenessProbe

doesn't get a response. Since replaying 9 hours of journals can take 30+ minutes, you need to adjust your Helm chart:

Copy code

yaml
# In your values.yaml or FE spec
livenessProbe:
  initialDelaySeconds: 300 # Increase to 5-10 minutes
  periodSeconds: 30
  failureThreshold: 60    # Increase threshold so it allows for 30+ mins of startup
readinessProbe:
  initialDelaySeconds: 300
  periodSeconds: 30
  failureThreshold: 60

Alternatively, you can temporarily remove the probes from the deployment just to let the recovery finish. Step 3: Check the "Last Words" Check the logs of the container that just crashed:

Copy code

bash
kubectl logs starrocks-fe-0 -p

If you see an error like

java.lang.OutOfMemoryError: Java heap space

, then the OOMKilled diagnosis is confirmed. You will need to give the FE more memory to handle the large metadata volume. Summary of Action Plan: 1. Increase FE Memory: If it's an OOM, double the RAM. 2. Extend Probes: Set

failureThreshold

to a very high number (e.g., 100) to give the FE 1 hour to finish replaying logs without being killed. 3. Keep the Flag: Keep

bdbje_reset_election_group = true

until the FE successfully starts and you can run a query. Only then should you remove it. Do not worry about the
Miss meta block [20001]
warning; it is normal and not what is causing the restart. The restart is happening because the FE is taking too long to finish its internal work. References • administration/Meta_recovery.md References • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 8:02 AM

@Rocky Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: fe-meta: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: fe-meta-starrocks-fe-0 ReadOnly: false fe-log: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: fe-log-starrocks-fe-0 ReadOnly: false starrocks-fe-cm: Type: ConfigMap (a volume populated by a ConfigMap) Name: starrocks-fe-cm Optional: false starrocks-fe-entrypoint-script-3907: Type: ConfigMap (a volume populated by a ConfigMap) Name: starrocks-fe-entrypoint-script Optional: false risingwave-hmac-secret-9572: Type: Secret (a volume populated by a Secret) SecretName: risingwave-hmac-secret Optional: false QoS Class: Burstable Node-Selectors: cloud.google.com/gke-nodepool=analytics Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m38s default-scheduler Successfully assigned trino/starrocks-fe-0 to gke-cortex-dev-eu1-analytics-7e367b4c-c9ti Normal Killing 4m29s kubelet Container fe failed startup probe, will be restarted Normal Pulled 4m28s (x2 over 9m29s) kubelet Container image "gcr.io/medable-registry/risingwave-starrocks-fe:1.1.1" already present on machine Normal Created 4m28s (x2 over 9m29s) kubelet Created container: fe Normal Started 4m28s (x2 over 9m29s) kubelet Started container fe Warning Unhealthy 4m24s (x61 over 9m24s) kubelet Startup probe failed: Get "10.4.12.23/api/health": dial tcp 10.4.12.238030 connect: connection refused kushagra@Kushagras-MacBook-Pro ~ %

Rocky

08/20/2026, 8:03 AM

The

Events

section confirms the issue: Kubernetes is killing your FE because the
startupProbe
is failing.

Copy code

text
Warning Unhealthy ... Startup probe failed: Get "<http://10.4.12.23:8030/api/health>": connection refused
Normal  Killing ... Container fe failed startup probe, will be restarted

Why this is happening: The StarRocks FE does not open its HTTP port (

) or its MySQL port (

) until after it has finished loading the metadata image and replaying all journal logs. Because you have 9 hours of logs to replay, the FE is taking longer than the

startupProbe

allows. Every time the probe fails enough times, Kubernetes kills the FE and starts the replay from zero again, putting you in an infinite loop. How to fix it: You need to give the FE much more time to "warm up" during this recovery. You have two options: Option 1: Increase the Startup Probe (Recommended) Modify your StarRocks FE deployment (or Helm

values.yaml

) to increase the

failureThreshold

. For example, to give it 1 hour to recover:

Copy code

yaml
startupProbe:
  httpGet:
    path: /api/health
    port: 8030
  periodSeconds: 10
  failureThreshold: 360 # 360 * 10 seconds = 60 minutes

Option 2: Temporarily Disable Probes (Fastest for recovery) If you want to fix this immediately without calculating times, you can edit the deployment directly and remove the

startupProbe

and

livenessProbe

sections: 1.

kubectl edit statefulset starrocks-fe

2. Scroll down to the

containers:

section. 3. Find and delete the

startupProbe

and

livenessProbe

blocks. 4. Save and exit. What to do next: Once you have extended or removed the probes: 1. The pod will start and stay running (no more

Killing

events). 2. Monitor the logs:

kubectl logs -f starrocks-fe-0

. 3. You will eventually see:

Finished to load image...

followed by thousands of lines saying

replayed journal id: XXXXX

. 4. Wait. Once the last journal is replayed, the FE will open port 8030/9030, and the pod will finally show as

READY 1/1

. Note: Since you are using the property

bdbje_reset_election_group = true

, once the FE is finally

READY

, remember to remove that property from your configuration so it doesn't try to reset the group every time it restarts in the future.

kushagra.dubey

08/20/2026, 8:06 AM

@Rocky, these are the values currently -

Copy code

startupProbe:
          failureThreshold: 60
          httpGet:
            path: /api/health
            port: 8030
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1

Rocky

08/20/2026, 8:06 AM

Your current configuration allows for only 5 minutes (60 failures × 5 seconds) before Kubernetes kills the process. For a recovery involving 9 hours of journals, 5 minutes is almost certainly not enough time. The FE is likely getting killed right in the middle of replaying those logs. The Fix You should increase the

failureThreshold

significantly. Give it at least 30 to 60 minutes to be safe. Since your

periodSeconds

is 5, you should set the threshold to 720 (for 1 hour). Update your configuration to this:

Copy code

yaml
startupProbe:
  failureThreshold: 720  # 720 * 5s = 3600s (1 hour)
  httpGet:
    path: /api/health
    port: 8030
    scheme: HTTP
  periodSeconds: 5
  successThreshold: 1
  timeoutSeconds: 5      # Increased slightly to be safe

Why this is necessary: 1. Metadata Loading: The FE first reads the

image

file into memory. It does not open port 8030 during this time. 2. Journal Replay: After loading the image, it must replay every transaction from the last 9 hours. If you have a high-traffic cluster, this could be hundreds of thousands of entries. 3. Port Opening: The FE only starts listening on port 8030 after the replay is 100% finished. If Kubernetes kills the FE at minute 6, all the progress it made replaying those journals is lost, and it starts from the beginning again on the next restart. This is why you are seeing the restart loop. How to verify progress after applying the change: Once you increase the threshold and the pod stops restarting, run:

Copy code

bash
kubectl logs -f starrocks-fe-0

You should see it eventually move past the "Miss meta block" warning and start printing lines like:

replayed journal id: 12345...

As long as those IDs are increasing, the FE is healthy and working. Do not manual restart it; just let it finish the replay until it finishes.

kushagra.dubey

08/20/2026, 8:15 AM

@Rocky, after this change these are the only logs I am seeing for last 5 minutes - ==> /opt/starrocks/fe/log/fe.out <== [2026-08-20 080822] notify new FE type transfer: LEADER ==> /opt/starrocks/fe/log/fe.log <== 2026-08-20 080832.327Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [ColocateTableIndex.cleanupInvalidDbOrTable():940] remove 0 invalid tableid: [] 2026-08-20 080832.597Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.loadImage():1699] Miss meta block [20001], This may not be a fatal error. It may be because there are new features in the version you upgraded this time, but there is no relevant metadata. 2026-08-20 080941.645Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 081041.650Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 081141.656Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 081241.675Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details. 2026-08-20 081341.691Z WARN (UNKNOWN starrocks-fe-0.starrocks-fe-search.trino.svc.cluster.local_9010_1778607334221(-1)|1) [GlobalStateMgr.waitForReady():1271] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many BDB logs to replay, because of previous failure of checkpoint(you can check the create time of image file under meta/image dir). 2. Majority voting members(LEADER or FOLLOWER) of the FE cluster haven't started completely. 3. FE node has multiple IPs, you should configure the priority_networks in fe.conf to match the ip record in meta/image/ROLE. And we don't support change the ip of FE node. Ignore this reason if you are using FQDN. 4. The time deviation between FE nodes is greater than 5s, please use ntp or other tools to keep clock synchronized. 5. The configuration of edit_log_port has changed, please reset to the original value. 6. The replayer thread may get stuck, please use jstack to find the details.

Rocky

08/20/2026, 8:15 AM

This is actually good news. The pod is no longer restarting, which means the

startupProbe

change worked. You have successfully broken the restart loop. The warnings you are seeing now:

It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER)

...are expected when you have a massive amount of journal logs (9 hours) to replay. The FE is currently in a "Leader" role but it is not yet "Ready" because it is still busy reconstructing the database state in memory from those logs. Is it actually making progress? To confirm the FE isn't just stuck, you need to check if the Journal ID is increasing. 1. Check the logs for replay progress: Run this command and look for "replayed journal id":

Copy code

bash
    kubectl logs starrocks-fe-0 | grep "replayed journal id"

Note: StarRocks usually logs this every 1,000 to 10,000 entries. If you don't see it immediately, wait a minute and check again. 2. Check CPU Usage:

Copy code

bash
    kubectl top pod starrocks-fe-0

If CPU usage is high (e.g., 100% of a core or more), the FE is actively replaying logs. Why it takes so long: * 9 hours of journals: Depending on your cluster's write load, this could be millions of edit log entries. * Single-threaded replay: Replaying these journals into the memory state is largely a single-threaded process. * Safe State: The FE will not open the MySQL port (

) or return

OK

to health checks until the very last journal entry is processed. What you should do: 1. Wait. Do not restart the pod. Every time you restart it, it has to start replaying those 9 hours of logs from the very beginning. 2. Monitor the logs. As long as you don't see

OutOfMemoryError

or a crash, it is working. 3. Check the Image file: Look at the timestamp of the image file in the metadata directory if you can (via

kubectl exec

). If it's very old, it confirms that the FE has a lot of "catching up" to do because a checkpoint hasn't happened recently. How will you know it's done? The warnings will stop, and you will see a log entry saying:

finished to open edit log, current max journal id: XXXXX

At that point, the

startupProbe

will finally succeed, the pod will show

1/1 READY

, and you can connect via MySQL. Once it is READY: Remember to remove

bdbje_reset_election_group = true

from your configuration so that it doesn't trigger a group reset on the next normal restart. References • administration/Meta_recovery.md • deployment/deploy_manually.md References • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 8:19 AM

@Rocky, this is the latest log after 10 minutes - at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-08-20 081858.573Z WARN (thrift-server-pool-94|1859) [TIOStreamTransport.close():153] Error closing output stream. java.net.SocketException: Socket closed at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(NioSocketImpl.java:165) ~[?:?] at sun.nio.ch.NioSocketImpl.beginWrite(NioSocketImpl.java:371) ~[?:?] at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:416) ~[?:?] at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?] at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?] at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?] at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-08-20 081901.393Z WARN (JournalWriter|190) [JournalWriter.updateBatchMetrics():227] slow edit log write, batch size: 1, took: 2027ms, current journal queue size: 0, please check the IO pressure of FE LEADER node or the latency between LEADER and FOLLOWER nodes

Rocky

08/20/2026, 8:19 AM

The logs you are seeing now indicate two things: your FE has finally become the Leader, but it is struggling with extremely slow disk I/O. 1. "slow edit log write" (The Root Cause)

Copy code

text
slow edit log write, batch size: 1, took: 2027ms

This is the most critical log. It means it took over 2 seconds just to write a single entry to the metadata journal. * Why this matters: StarRocks FE metadata is persistent. Every time the Leader does something (like starting up, cleaning up, or handling a request), it must write to this log. * The Impact: Since you are replaying 9 hours of journals, if the disk is this slow, the "finalizing" stage of the startup will take a very long time. This is likely why the FE hasn't opened port

yet—it's stuck waiting for disk writes to complete during its initialization. * Check your Disk: Since you are on GKE, check if you are using

pd-standard

(HDD) or

pd-ssd

. For StarRocks metadata, SSD is highly recommended. If the disk is under heavy "IO Pressure," the FE will appear hung. 2. "SocketException: Socket closed" These Thrift errors are usually secondary. They happen because: * The FE has started its internal RPC services, and other components (like your BEs or external monitoring) are trying to connect. * Because the FE is so bogged down by the slow disk writes (the 2-second lag), it is failing to respond to those connections in time, or the connections are timing out and being closed by the other side. What you should do now: 1. Do not restart: You have already passed the "election" phase (it is now

LEADER

). If you restart now, you will lose the progress it made and start the slow BDB election/replay all over again. 2. Monitor the progress: Check the number of files in the metadata

journal

directory:

Copy code

bash
    kubectl exec starrocks-fe-0 -- ls -l /opt/starrocks/fe/meta/bdb

If you see the timestamps updating or new files appearing, the FE is alive and moving, just very slowly. 3. Check for "Ready" status: The FE will only open port

(passing the

startupProbe

) after it finishes these initial slow writes and fully initializes its internal state. 4. Check for CPU/Memory exhaustion:

Copy code

bash
    kubectl top pod starrocks-fe-0

If memory is near the limit, it might be swapping or performing heavy Garbage Collection, which also slows down disk I/O. Summary: Your FE is no longer in a restart loop, but it is "limping" due to slow disk performance. It should eventually become ready, but the 2-second write latency is a serious bottleneck. If this is a production cluster, you may need to upgrade the underlying PV to a faster disk type (SSD) once you get the cluster back online. References • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md

kushagra.dubey

08/20/2026, 8:22 AM

@Rocky, okay so currently we only have 1 pod (since we removed the replicas) and CPU is 4, and I could see that used was crossing the limits

Rocky

08/20/2026, 8:22 AM

If your CPU usage is crossing the limits, you have identified the main bottleneck. When a Kubernetes pod hits its CPU limit, the OS "throttles" the process (literally pauses it for milliseconds at a time). For a Java process like the StarRocks FE, this is devastating because: 1. Thread Stuttering: The "slow edit log write" warning (2027ms) is likely not the disk's fault. It’s because the thread trying to write the log is being paused by the Kubernetes scheduler mid-operation. 2. RPC Timeouts: The

SocketException

happens because the FE is too "starved" of CPU to respond to internal network heartbeats, causing other components to drop the connection. 3. Slow Replay: Replaying 9 hours of journals involves heavy Java object de-serialization, which is very CPU-intensive. Recommended Action: You should increase the CPU limit immediately. * Temporary Fix for Recovery: Bump the CPU limit to 8 or 12 cores if your node has the capacity. This will allow the FE to "burst" and finish replaying those journals much faster. * Guaranteed Resources: Set your CPU

requests

equal to your

limits

. This prevents other pods on the same node from "stealing" cycles that the FE needs for metadata integrity. Example Update:

Copy code

yaml
resources:
  requests:
    cpu: "8"
    memory: "16Gi" # Ensure memory is also sufficient
  limits:
    cpu: "8"
    memory: "16Gi"

Why 4 CPUs isn't enough right now: While 4 CPUs might be enough for a "quiet" cluster, during a massive metadata recovery (like replaying 9 hours of logs), the FE has to do a huge amount of work to reconstruct the state in memory. Summary: By hitting the CPU limit, you are making a 1-hour recovery process take 5+ hours (or never finish). Increase the CPU, and you will see those "slow edit log write" warnings disappear, and the FE should become

READY

shortly after. Check for Throttling: You can verify this by running:

Copy code

bash
kubectl describe pod starrocks-fe-0

Look for "Terminated" reasons or check the node's metrics to see if

container_cpu_cfs_throttled_seconds_total

is climbing. References • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md

kushagra.dubey

08/20/2026, 8:25 AM

@Rocky, okay but we still don't add replicas, only increase resources of 1 active pod?

Rocky

08/20/2026, 8:26 AM

Yes, keep it at exactly 1 pod for now. Do not add replicas yet. Here is why you should focus only on the single active pod: 1. Metadata Authority: By using

bdbje_reset_election_group = true

, you have told this specific pod: "Forget about all other nodes; you are the only source of truth." It is currently rebuilding the entire database state from the journal. 2. Overhead: If you add replicas now, the Leader (which is already struggling and CPU-throttled) would have to spend extra resources trying to communicate with and replicate data to those new pods. This will only slow down the recovery. 3. Clean Slate: When you eventually do add replicas, they must start with a completely empty metadata directory so they can perform a "Remote Bootstrap" (download the clean metadata from your current Leader). The Recommended Path: Step 1: Increase resources for the 1 pod Give it as much CPU and Memory as your K8s nodes can spare (e.g., 8 or 16 CPUs). This will stop the "slow edit log write" errors and speed up the 9-hour journal replay. Step 2: Wait for "1/1 READY" Wait until the logs show:

finished to open edit log, current max journal id: XXXXX

And the pod status becomes

READY

. At this point, your data is safe and the FE is functional. Step 3: Remove the Recovery Flag 1. Edit your configuration to remove

bdbje_reset_election_group = true

. 2. Restart the single pod. It will start up quickly this time because the journals have already been replayed. Step 4: Scale up to 3 replicas Only after the first pod is stable and the recovery flag is gone should you scale the replicas back to 3. * Note: If you are using the StarRocks Operator, it should handle the joining process. If you are doing it manually, ensure the new pods are started with the

--helper

flag pointing to the first pod's IP. Summary: For now, more CPU for the single pod is the fastest way to get your cluster back online. Adding more pods now will only complicate the recovery. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/helm_chart_usage.md • administration/Meta_recovery.md References • administration/management/FE_configuration.md • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 8:31 AM

@Rocky, I will do this change from helm charts which can overritw the startupprobe change, can I add the starupprobe change in helm as well

Rocky

08/20/2026, 8:31 AM

Yes, you can and should manage both the CPU resources and the startupProbe through your Helm

values.yaml

file. This is the best way to ensure the changes persist and that the Operator doesn't revert your manual

kubectl edit

changes. Here is the structure you should use in your Helm values file (assuming you are using the official

kube-starrocks

chart): Example
values.yaml
for Recovery

Copy code

yaml
starrocks:
  starrocksFESpec:
    replicas: 1  # Keep at 1 until recovery is finished
    resources:
      requests:
        cpu: "8"      # Increase this to stop throttling
        memory: "16Gi" 
      limits:
        cpu: "8"
        memory: "16Gi"
    
    # Increase the startupProbe timeout to prevent K8s from killing the pod 
    # while it is replaying the 9-hour journal.
    startupProbe:
      httpGet:
        path: /api/bootstrap
        port: 8030
      periodSeconds: 10
      failureThreshold: 300 # 10s * 300 = 50 minutes of allowed startup time
      
    # Ensure your recovery flag is still present here
    configParameters:
      bdbje_reset_election_group: "true"

Why these values? 1. CPU Requests = Limits: In GKE/K8s, setting these equal ensures the pod gets "Guaranteed" QoS, meaning other pods on the node cannot steal the CPU cycles your FE needs for its heavy metadata replay. 2. failureThreshold: Setting this to a high number (like 300 or 600) gives the FE a massive window to finish its work without being restarted. Even if the FE is "stuck" for 30 minutes replaying logs, K8s will wait patiently as long as the probe hasn't exceeded the threshold. How to apply: Run your helm upgrade with the new values:

Copy code

bash
helm upgrade <release_name> starrocks/kube-starrocks -f values.yaml

Warning: Once the FE is finally

READY

and you can log in via MySQL, remember to: 1. Set

bdbje_reset_election_group: "false"

(or remove it). 2. Upgrade the chart again. 3. Only then, scale

replicas

back to 3. References • deployment/helm_chart_usage.md • quick_start/helm.md • administration/Meta_recovery.md • administration/management/FE_configuration.md References • quick_start/helm.md • deployment/sr_operator.md

Rocky

08/20/2026, 9:34 AM

If you are seeing the pod restart even after increasing the CPU and the startup probe, and the

textPayload

in your logs is empty, we need to look at how the container is dying. An empty

textPayload

often happens when the process is killed by the OS (Kubernetes) before it can even write a log entry about why it is failing. Please check these three things immediately: 1. Check for
OOMKilled
(Out of Memory) Replaying 9 hours of metadata is extremely memory-intensive. If your memory limit is too low, the Linux kernel will kill the process (OOMKill). * Run this:

kubectl get pod <fe-pod-name> -o yaml

* Look for:

lastState: terminated

and check if the

reason

OOMKilled

. * Fix: If it says

OOMKilled

, increase your memory limit in Helm to at least 16Gi or 32Gi for this recovery period. 2. Is the
livenessProbe
killing it? In your Helm chart, you likely have both a

startupProbe

and a

livenessProbe

. * If the

startupProbe

finishes, but the FE is still doing heavy background work and doesn't respond to the

livenessProbe

quickly enough, Kubernetes will restart it. * Action: Increase the

failureThreshold

initialDelaySeconds

for your livenessProbe as well, or disable it temporarily during the recovery. 3. Check the JVM Heap configuration If you increased the Pod's memory limit but didn't increase the FE Java Heap size, the FE will crash with an "Out of Memory Error" internally. * Check your

fe.conf

(or the

configParameters

in Helm). * Ensure

-Xmx

is set to about 80% of your pod's memory limit. _ _Example:* If the pod has 16Gi memory, set

JAVA_OPTS

to include

-Xmx12g

. 4. BDBJE Lock Files If the FE crashed hard, sometimes a

je.lck

file is left in the

meta/bdb

directory. Usually, the FE can override this on restart, but in rare cases, it prevents startup. * Check the

fe.warn.log

fe.log

inside the pod (if you can

exec

in before it restarts) for any mention of

LockTimeoutException

File not found

. Next Step to diagnose: Since the GCP log you shared is empty, please run this command to see the actual reason Kubernetes stopped the last container:

Copy code

bash
kubectl describe pod <fe-pod-name>

Look at the "Containers" section, specifically the "Last State" and "Exit Code". * Exit Code 137: Means OOMKilled or SIGKILL (K8s killed it). * Exit Code 143: Means SIGTERM (K8s stopped it gracefully, likely a probe failure). * Exit Code 1: Means the Java process crashed itself (check

fe.log

). What did the
Last State
say in the
kubectl describe
output? References • administration/management/FE_configuration.md • administration/Meta_recovery.md • deployment/helm_chart_usage.md • administration/management/monitoring/metrics.md References • administration/management/FE_configuration.md • administration/Meta_recovery.md • administration/management/monitoring/metrics.md

kushagra.dubey

08/20/2026, 9:34 AM

@Rocky, restarts are still happennig after memory increase and I am seeing one more log - { "textPayload": "java.lang.OutOfMemoryError: Java heap space", "insertId": "8qfal97edel0v7ri", "resource": { "type": "k8s_container", "labels": { "location": "europe-west3", "container_name": "fe", "project_id": "medable-cortex", "namespace_name": "trino", "cluster_name": "cortex-dev-eu1", "pod_name": "starrocks-fe-0" } }, "timestamp": "2026-08-20T093204.504011673Z", "severity": "INFO", "labels": { "k8s-pod/apps_kubernetes_io/pod-index": "0", "k8s-pod/app_starrocks_ownerreference/name": "starrocks-fe", "logging.gke.io/top_level_controller_type": "StatefulSet", "compute.googleapis.com/resource_name": "gke-cortex-dev-eu1-analytics-7e367b4c-xst6", "k8s-pod/app_kubernetes_io/component": "fe", "logging.gke.io/top_level_controller_name": "starrocks-fe", "k8s-pod/controller-revision-hash": "starrocks-fe-7d5898d4d9", "k8s-pod/statefulset_kubernetes_io/pod-name": "starrocks-fe-0" }, "logName": "projects/medable-cortex/logs/stdout", "receiveTimestamp": "2026-08-20T093205.371791139Z" }

Rocky

08/20/2026, 9:35 AM

This error confirms that while you increased the Pod's memory, the Java Virtual Machine (JVM) inside the Pod is still restricted to a smaller "Heap" size. By default, the FE might only be using 4GB or 8GB of Heap, regardless of how much memory the Pod has. To fix this, you must explicitly tell Java to use more of the available Pod memory. How to fix it in Helm: Update your

values.yaml

to include

JAVA_OPTS

inside the

configParameters

. If your Pod has 16Gi of memory, set the Heap (

-Xmx

) to about 12Gi (roughly 75-80% of the total).

Copy code

yaml
starrocks:
  starrocksFESpec:
    resources:
      requests:
        cpu: "8"
        memory: "16Gi"
      limits:
        cpu: "8"
        memory: "16Gi"
    
    configParameters:
      # This is the critical line to fix the OutOfMemoryError
      JAVA_OPTS: "-Xmx12g -Xms12g -XX:+UseG1GC"
      
      # Keep your recovery flag active
      bdbje_reset_election_group: "true"

Why this is happening: 1. Metadata in RAM: StarRocks keeps all metadata in memory. Replaying 9 hours of journals means the FE is trying to load a massive amount of data into the JVM Heap at once. 2. Container vs. JVM: Kubernetes limits the container's memory, but the Java process has its own internal limit (

-Xmx

). If

-Xmx

is not set, Java often defaults to a fraction of the available memory, which isn't enough for a large metadata recovery. What to check next: 1. Apply the change:

helm upgrade <release> ...

2. Verify the JVM setting: Once the pod starts, you can verify it's using the new limit by running:

Copy code

bash
    kubectl exec starrocks-fe-0 -- ps aux | grep java

Look for

-Xmx12g

in the command output. 3. Monitor the logs: You should see the journal replay progress further now. If it crashes with OOM again even at 12GB, you may need to increase the Pod to 32Gi and set

JAVA_OPTS

-Xmx24g

. Important: If the journal replay takes a long time, ensure your

startupProbe.failureThreshold

is still set high (e.g., 600) so Kubernetes doesn't kill the pod while it's busy loading the metadata. References • faq/fe_mem_faq.md • administration/management/FE_configuration.md • faq/Others.md • administration/Meta_recovery.md References • administration/management/FE_configuration.md • faq/Others.md • administration/Meta_recovery.md • faq/fe_mem_faq.md References • quick_start/helm.md • sql-reference/sql-functions/JAVA_UDF.md • faq/Others.md

kushagra.dubey

08/20/2026, 9:37 AM

@Rocky, this is what I have in my helm charts - if [ -d "${META_DIR}" ] && [ "$(ls -A ${META_DIR} 2>/dev/null)" ]; then echo "[$(date '+%a %b %d %T %Z %Y')] start fe with exist meta." else echo "[$(date '+%a %b %d %T %Z %Y')] start fe without exist meta." fi echo "[$(date '+%a %b %d %T %Z %Y')] start run start_fe.sh with options: '${EXTRA_OPTS}'" export JAVA_OPTS="-Xmx5g -Xms2g" mkdir -p ${STARROCKS_HOME}/fe/log touch ${STARROCKS_HOME}/fe/log/fe.log > ${STARROCKS_HOME}/fe/log/fe.out ${STARROCKS_HOME}/fe/bin/start_fe.sh ${EXTRA_OPTS} & FE_PID=$! tail -fn 0 ${STARROCKS_HOME}/fe/log/fe.out ${STARROCKS_HOME}/fe/log/fe.log & wait ${FE_PID} exit $?

Rocky

08/20/2026, 9:37 AM

That line in your script is the exact cause of the crash. Even though you increased the Pod's memory to 16Gi in Kubernetes, your script has hardcoded the Java Heap to a maximum of 5GB:

export JAVA_OPTS="-Xmx5g -Xms2g"

The FE is trying to replay 9 hours of journals, which requires loading a lot of metadata into RAM. When it hits that 5GB limit, it throws the

OutOfMemoryError

and restarts. How to fix it: You need to edit your Helm chart's command/script to use more of the available 16Gi. Change that line to:

Copy code

bash
export JAVA_OPTS="-Xmx12g -Xms12g"

Why these values? 1. -Xmx12g: This gives the FE 12GB of heap space. Since your Pod limit is 16Gi, this leaves ~4GB for the operating system and non-heap memory, which is safe. 2. -Xms12g: Setting the starting memory (

Xms

) equal to the maximum (

Xmx

) is a StarRocks best practice. It prevents the JVM from wasting time resizing the memory while it's trying to recover your metadata. Recommendation for Recovery: If 12GB still fails (replaying 9 hours of journals is very heavy), I strongly recommend bumping the resources even higher just for this recovery phase: * Pod Memory Limit:

32Gi

* JAVA_OPTS:

"-Xmx24g -Xms24g"

Once the FE is back online and "Healthy," you can scale these back down to your normal production values. Don't forget: After you update the Helm chart and the FE starts, check the

fe.log

to confirm it is successfully "replaying" the journal entries. You should see progress messages showing the journal IDs being processed. References • faq/fe_mem_faq.md • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/Others.md References • knowledge/trouble_shooting/fe_memory_problems.md • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 9:41 AM

@Rocky, these are the few more errors in the logs - uster.local) 22601387(starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local) timeout=20s 2026-08-20 093954.295Z WARN (thrift-server-pool-10|1602) [FrontendServiceImpl.createPartitionProcess():2257] failed to add partitions com.starrocks.common.DdlException: Table creation timed out. unfinished replicas(2/2): 22601406(starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local) 22601387(starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local) timeout=20s You can increase the timeout by increasing the config "tablet_create_timeout_second" and try again. To increase the config "tablet_create_timeout_second" (currently 10), run the following command:

Copy code

admin set frontend config("tablet_create_timeout_second"="20")

or add the following configuration to the fe.conf file and restart the process:

Copy code

tablet_create_timeout_second=20

at com.starrocks.task.TabletTaskExecutor.waitForFinished(TabletTaskExecutor.java:423) ~[starrocks-fe.jar:?] at com.starrocks.task.TabletTaskExecutor.sendCreateReplicaTasksAndWaitForFinished(TabletTaskExecutor.java:309) ~[starrocks-fe.jar:?] at com.starrocks.task.TabletTaskExecutor.buildPartitionsSequentially(TabletTaskExecutor.java:111) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.buildPartitions(LocalMetastore.java:2012) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.addPartitions(LocalMetastore.java:1358) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.addPartitions(LocalMetastore.java:971) ~[starrocks-fe.jar:?] at com.starrocks.service.FrontendServiceImpl.createPartitionProcess(FrontendServiceImpl.java:2251) ~[starrocks-fe.jar:?] at com.starrocks.service.FrontendServiceImpl.createPartition(FrontendServiceImpl.java:2086) ~[starrocks-fe.jar:?] at com.starrocks.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:6705) ~[starrocks-fe.jar:?] at com.starrocks.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:6682) ~[starrocks-fe.jar:?] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-08-20 093959.386Z WARN (thrift-server-pool-11|1608) [TabletTaskExecutor.waitForFinished():408] Table creation timed out. unfinished replicas(1/1): 22609910(starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local) timeout=10s 2026-08-20 093959.386Z WARN (thrift-server-pool-11|1608) [FrontendServiceImpl.createPartitionProcess():2257] failed to add partitions com.starrocks.common.DdlException: Table creation timed out. unfinished replicas(1/1): 22609910(starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local) timeout=10s You can increase the timeout by increasing the config "tablet_create_timeout_second" and try again. To increase the config "tablet_create_timeout_second" (currently 10), run the following command:

Copy code

admin set frontend config("tablet_create_timeout_second"="20")

or add the following configuration to the fe.conf file and restart the process:

Copy code

tablet_create_timeout_second=20

at com.starrocks.task.TabletTaskExecutor.waitForFinished(TabletTaskExecutor.java:423) ~[starrocks-fe.jar:?] at com.starrocks.task.TabletTaskExecutor.sendCreateReplicaTasksAndWaitForFinished(TabletTaskExecutor.java:309) ~[starrocks-fe.jar:?] at com.starrocks.task.TabletTaskExecutor.buildPartitionsSequentially(TabletTaskExecutor.java:111) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.buildPartitions(LocalMetastore.java:2012) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.addPartitions(LocalMetastore.java:1358) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.addPartitions(LocalMetastore.java:971) ~[starrocks-fe.jar:?] at com.starrocks.service.FrontendServiceImpl.createPartitionProcess(FrontendServiceImpl.java:2251) ~[starrocks-fe.jar:?] at com.starrocks.service.FrontendServiceImpl.createPartition(FrontendServiceImpl.java:2086) ~[starrocks-fe.jar:?] at com.starrocks.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:6705) ~[starrocks-fe.jar:?] at com.starrocks.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:6682) ~[starrocks-fe.jar:?] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-08-20 093959.454Z WARN (thrift-server-pool-14|1626) [DefaultCoordinator.updateRuntimeProfile():1245] exec state report failed status=errorCode RUNTIME_ERROR automatic create partition failed. error:Table creation timed out. unfinished replicas(1/1): 22609910(starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local) timeout=10s You can increase the timeout by increasing the config "tablet_create_timeout_second" and try again.

Rocky

08/20/2026, 9:41 AM

These new errors indicate that while your FE is now staying up, it is struggling to coordinate with your CN (Compute Nodes) to create the actual storage structures (tablets/partitions). This is very common during a recovery because once the FE is back, it often tries to "catch up" on many tasks at once, leading to congestion. 1. Increase the Timeout Immediately You have already reached the 10s (and even 20s) limit. Increase

tablet_create_timeout_second

to 60 or 120 seconds to give the system breathing room during this recovery phase. In your Helm

values.yaml

, add this to the

configParameters

section of your FE:

Copy code

yaml
starrocksFESpec:
  configParameters:
    # ... your other configs ...
    tablet_create_timeout_second: "120"

2. Check the Health of your CN Pods The logs specifically mention

starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local

as the node failing to respond. If the CNs are overwhelmed or restarting, the FE cannot finish creating the partitions. * Check Pod Status:

kubectl get pods -n trino -l <http://app.kubernetes.io/component=cn|app.kubernetes.io/component=cn>

* Check CN Logs:

kubectl logs -f starrocks-cn-2 -n trino

* Look for "Out of Memory" errors or high CPU usage in the CN logs. If the CNs are also crashing, you may need to increase their memory limits as well. 3. Verify CN Registration Since you just recovered the FE metadata, ensure the FE actually "sees" the CNs as alive. Connect to the FE via MySQL and run:

Copy code

sql
SHOW COMPUTE NODES;

Look at the

Alive

column. If it says

false

for any node, the FE won't be able to create tablets on them. Why this is happening now: * Backlog: While the FE was down for 9 hours, any "Routine Load" or "Insert" jobs were queued. Now they are all hitting the FE at once. * Dynamic Partitioning: If your tables use dynamic partitioning, the FE is likely trying to create dozens of partitions for the "missed" time windows all at once. * Resource Contention: If your CNs are also on the same GKE nodes as the FE, the heavy journal replay might have starved them of resources. Summary of Action: 1. Set

tablet_create_timeout_second: "120"

in your Helm FE config. 2. Restart/Update the FE. 3. Verify the CNs are healthy and

Alive: true

SHOW COMPUTE NODES

. References • administration/management/FE_configuration.md • faq/Sql_faq.md • faq/loading/Routine_load_faq.md • administration/management/BE_configuration.md References • administration/management/FE_configuration.md • faq/loading/Routine_load_faq.md • faq/Sql_faq.md References • administration/management/FE_configuration.md • faq/Sql_faq.md • faq/loading/Routine_load_faq.md • administration/management/BE_configuration.md References • administration/management/FE_configuration.md • faq/Sql_faq.md • administration/management/BE_configuration.md • faq/loading/Routine_load_faq.md • faq/shared_data_faq.md

kushagra.dubey

08/20/2026, 1:15 PM

The FE pod came up, but now how can I switch back to 3 fe pod setup @Rocky

Rocky

08/20/2026, 1:16 PM

To switch back to a 3-FE High Availability (HA) setup, you must follow these steps carefully. The most important part is removing the recovery flag you used to get the first pod started. Step 1: Verify the Current FE is Leader Log into your StarRocks cluster (via MySQL) and run:

Copy code

sql
SHOW FRONTENDS;

Confirm that

starrocks-fe-0

(or whichever pod is up) has

Role: LEADER

and

Alive: true

. Because you used

bdbje_reset_election_group: "true"

, this FE has "forgotten" about the other two FEs. Step 2: Prepare for Scaling in Helm Update your

values.yaml

. You need to do two things at once: 1. Set
replicas
back to 3. 2. Remove or set
bdbje_reset_election_group
to
"false"
. If you leave it as

true

, the pods will constantly try to reset the group and won't be able to form a 3-node quorum.

Copy code

yaml
starrocks:
  starrocksFESpec:
    replicas: 3  # Scale back to 3
    configParameters:
      bdbje_reset_election_group: "false" # CRITICAL: Set to false
      JAVA_OPTS: "-Xmx12g -Xms12g"        # Keep the high memory
      tablet_create_timeout_second: "120" # Keep the higher timeout

Step 3: Clear Old Metadata for the New FEs (Highly Recommended) Since you performed a "hard reset" on the leader's metadata group, the old data sitting on the disks (PVCs) of

fe-1

and

fe-2

is now technically from a "different" timeline. They will likely fail to join

fe-0

if they try to use their old data. Before you run the Helm upgrade, delete the PVCs for the other two FEs so they start completely fresh and sync from the Leader:

Copy code

bash
# Verify the PVC names first
kubectl get pvc -n trino

# Delete the PVCs for FE 1 and 2 (the operator will recreate them)
kubectl delete pvc <metadata-pvc-name-for-fe-1> -n trino
kubectl delete pvc <metadata-pvc-name-for-fe-2> -n trino

Step 4: Apply the Changes Run your Helm upgrade:

Copy code

bash
helm upgrade <release-name> starrocks/starrocks -f values.yaml -n trino

Step 5: Monitor the Join Process 1. FE-0 will restart: Since you changed the

bdbje_reset_election_group

parameter, the operator will restart

fe-0

. It should come back up as LEADER again. 2. FE-1 and FE-2 will start: They will see they have no data (because you cleared the PVCs), and they will automatically contact

fe-0

to join the cluster. 3. Check Status: Wait a few minutes and run:

Copy code

sql
    SHOW FRONTENDS;

You should eventually see all 3 FEs with

Alive: true

, one

LEADER

, and two `FOLLOWER`s. Summary of why these steps matter: * Flag Removal: If

bdbje_reset_election_group

stays

true

, the cluster can never have more than 1 node. * PVC Deletion: If you don't clear the old PVCs for

fe-1/2

, you will likely see

HANDSHAKE_ERROR

in the logs because their internal BDBJE "Node ID" doesn't match the new group created by your reset. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/sr_operator.md • quick_start/helm.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/sr_operator.md • quick_start/helm.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md • deployment/sr_operator.md

Rocky

08/20/2026, 1:16 PM

• quick_start/helm.md • administration/Cluster_administration.md References • administration/management/FE_configuration.md • administration/Meta_recovery.md

kushagra.dubey

08/20/2026, 1:19 PM

One issue which is happenig in current setup (1 Fe leader) is that starrocks pods are crashing both CN and FE @Rocky

Rocky

08/20/2026, 1:19 PM

The simultaneous crashing of FE and CN pods usually indicates a cascading resource failure caused by the 9-hour backlog you are replaying. When the FE finally comes up, it immediately tries to catch up on thousands of tasks (Routine Loads, Dynamic Partitioning, and Tablet Reports). This creates a "thundering herd" effect where the FE and CNs starve each other of resources. 1. Check for OOMKill (The most likely cause) Run this command to see why the pods actually stopped:

Copy code

bash
kubectl get pods -n trino -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,LAST_STATE:.status.containerStatuses[0].lastState.terminated.reason,EXIT_CODE:.status.containerStatuses[0].lastState.terminated.exitCode

* Reason: OOMKilled (Exit Code 137): The Pod exceeded its Kubernetes memory limit. * Reason: Error (Exit Code 1): The process crashed internally (check the logs for "OutOfMemoryError" or "Segmentation Fault"). 2. Why the CNs are crashing Your logs showed

starrocks-cn-2

failing to create tablets. If the CN is crashing with Exit Code 137 (OOMKilled): * Cause: The CN is likely trying to process a massive influx of data from Routine Loads that resumed as soon as the FE became leader. * Solution: Increase the CN memory limits in your Helm chart. If your FE needs 16Gi, your CNs (which do the actual heavy lifting) likely need 32Gi or 64Gi depending on your data volume. 3. Immediate Stability Strategy To stop the crashing and let the cluster stabilize, you should reduce the load until the metadata replay and tablet creation are finished: 1. Pause Routine Loads: If you can get a MySQL connection for even a minute, pause your heavy loading jobs:

Copy code

sql
    PAUSE ROUTINE LOAD FOR <job_name>;
    -- Or pause all if you have many:
    PAUSE ALL ROUTINE LOAD;

2. Increase CN Memory: Update your Helm chart to give CNs more RAM. StarRocks CNs are very memory-intensive.

Copy code

yaml
    starRocksCnSpec:
      requests:
        memory: "32Gi"
      limits:
        memory: "32Gi"

3. Check CN Logs for "Segmentation Fault": If the CN exit code is NOT 137, look at the end of the CN log (

kubectl logs starrocks-cn-2 -n trino

). If you see

Segmentation Fault

Signal 11

, it might be a specific bug related to the version or a corrupted tablet, and you should check the

cn.out

file inside the pod. 4. Check for Node-Level Pressure If both FE and CN are on the same physical GKE node, the FE's journal replay (which is CPU and I/O intensive) might be causing the CN to fail its Liveness/Readiness probes. * Check if the nodes themselves are under pressure:

kubectl top nodes

. * If the nodes are at 90%+ memory, Kubernetes will start killing pods to save the node. Summary of Action: 1. Confirm if it's

OOMKilled

using the

kubectl

command in Step 1. 2. If yes, double the memory for both FE and CN pods in your Helm chart. 3. Temporarily stop data ingestion (Routine Load/Stream Load) to let the FE finish its metadata tasks. References • administration/management/FE_configuration.md • faq/loading/Routine_load_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md

kushagra.dubey

08/20/2026, 1:44 PM

@Rocky, do we need to delete the logs PVCs as well

Rocky

08/20/2026, 1:45 PM

No, you do not need to delete the log PVCs (e.g.,

fe-log-...

). There is a big difference between "Metadata" and "Logs" in StarRocks: 1. Metadata PVC (Required to delete for followers) * Contents: BDBJE database files, journal logs (not text logs), and the FE image. * Why delete: This is where the cluster "identity" and data state live. Since you performed a metadata reset on the leader, the old metadata on

fe-1

and

fe-2

is now incompatible. They must start with an empty metadata directory to "handshake" correctly and download the new state from the leader. 2. Log PVC (Optional to delete) * Contents: Human-readable text files like

fe.log

fe.warn.log

, and

fe.audit.log

. * Why it's optional: These files are just for troubleshooting. They have no impact on whether an FE can join the cluster or not. * Recommendation: Keep them unless you are running out of disk space. They might contain useful history about why the pods were crashing earlier. Summary of Actions for FE-1 and FE-2: 1. Delete Metadata PVCs:

kubectl delete pvc <prefix>-meta-<pod-name>

— Mandatory. 2. Delete Log PVCs:

kubectl delete pvc <prefix>-log-<pod-name>

— Optional (only if you want a totally clean slate). Important Note on Terminology: Inside the

meta

directory, there is a folder often called

bdb

. BDBJE refers to its internal data files as "logs" (e.g.,

00000001.jdb

Rocky

08/20/2026, 1:45 PM

References • administration/management/logs.md • administration/management/FE_configuration.md • quick_start/helm.md

kushagra.dubey

08/20/2026, 2:23 PM

The 3 pod setup is up but I still see pod restarts happening due to this failure @Rocky

Copy code

2026-08-20 14:20:55.543Z WARN (ProcProfileCollector|62) [ProcProfileCollector.lambda$collectCPUProfile$2():128] collect cpu profile failed, reason: Profiler has not started
2026-08-20 14:20:56.947Z WARN (starrocks-http-pool-10|312) [HttpServerHandler$RequestHandlingWatch.finish():254] receive slow http request. uri: /api/bootstrap?cluster_id=*&token=*, startTime: 1787235645437, latency: 11510 ms
2026-08-20 14:20:56.947Z ERROR (ProcProfileCollector|62) [Daemon.run():100] daemon thread got exception. name: ProcProfileCollector
java.lang.RuntimeException: Failed to collect CPU profile
	at com.starrocks.memory.ProcProfileCollector.collectCPUProfile(ProcProfileCollector.java:129) ~[starrocks-fe.jar:?]
	at com.starrocks.memory.ProcProfileCollector.runAfterCatalogReady(ProcProfileCollector.java:61) ~[starrocks-fe.jar:?]
	at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?]
	at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?]
Caused by: java.lang.IllegalStateException: Profiler has not started
	at one.profiler.AsyncProfiler.execute0(Native Method) ~[async-profiler-4.0.jar:?]
	at one.profiler.AsyncProfiler.execute(AsyncProfiler.java:186) ~[async-profiler-4.0.jar:?]
	at com.starrocks.memory.ProcProfileCollector.collectCPUProfile(ProcProfileCollector.java:126) ~[starrocks-fe.jar:?]
	... 3 more
kushagra@Kushagras-MacBook-Pro ~ %

Rocky

08/20/2026, 2:24 PM

The

ProcProfileCollector

error you are seeing is likely a red herring. While it looks like a crash, it is a background diagnostic thread (new in v3.2+) that tries to collect CPU profiles for the Web UI. If it fails, it logs an error but typically doesn't crash the entire FE process. The actual cause of your restarts is revealed in the second log line:

receive slow http request. uri: /api/bootstrap ... latency: 11510 ms

Why your pods are restarting: 1. Liveness Probe Failure: The StarRocks Operator and Kubernetes use the

/api/bootstrap

(or

/api/health

) endpoint to check if the FE is alive. 2. Backlog Pressure: Because your FE is replaying a 9-hour metadata backlog, it is extremely busy and its internal locks are frequently held. This causes the HTTP server to respond very slowly (11.5 seconds in your log). 3. The Kill Switch: The default Kubernetes/Operator timeout for these checks is often 5 or 10 seconds. When the FE takes 11.5 seconds to respond, Kubernetes marks it as "Unhealthy" and kills the pod. 4. The Loop: The pod restarts, starts replaying the 9-hour backlog again, gets stuck/slow again, and gets killed again. You are in a "Replay-Restart Loop." How to fix it: 1. Increase Probe Timeouts (The Priority) You must give the FE more "breathing room" to finish its metadata replay. Update your Helm

values.yaml

to increase the liveness and readiness probe initial delays and timeouts:

Copy code

yaml
starrocksFESpec:
  livenessProbe:
    initialDelaySeconds: 300 # Wait 5 mins before starting checks
    periodSeconds: 30
    timeoutSeconds: 30      # Increase from 5s to 30s
    failureThreshold: 10    # Allow more consecutive failures
  readinessProbe:
    initialDelaySeconds: 300
    periodSeconds: 30
    timeoutSeconds: 30
    failureThreshold: 10

2. Disable the Background Profiler To reduce any unnecessary CPU overhead while the cluster is struggling, disable the feature causing that

ProcProfileCollector

error in your

fe.conf

(via

configParameters

in Helm):

Copy code

yaml
starrocksFESpec:
  configParameters:
    proc_profile_cpu_enable: "false"
    proc_profile_mem_enable: "false"

3. Monitor the Replay Progress Instead of looking at the Profiler error, look for these lines in your `fe.log`: *

replayed journal id: ...

catch up to journal id: ...

As long as the "replayed journal id" is increasing, the FE is working. If the pod restarts before the "replayed" number reaches the "catch up" number, you must increase the

timeoutSeconds

and

failureThreshold

further. Summary The

ProcProfileCollector

error is a secondary symptom of a stressed system. The Slow HTTP Request is what's triggering Kubernetes to kill your pods. Fix the probe settings to let the FE finish its work. References • administration/management/proc_profile.md • administration/management/FE_configuration.md • deployment/sr_operator.md • administration/Meta_recovery.md References • administration/management/monitoring/metrics.md • administration/http_interface.md • administration/management/FE_configuration.md • faq/Others.md • administration/management/BE_configuration.md • faq/Sql_faq.md

5 Views

Open in Slack

Previous Next