@Rocky Starrocks 4.1.3 is stuck in an endless reboot cycle. The logs show the following: │ log-tailer 2026-08-27 094245.734-04:00 WARN (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [StateChangeExecutor.notifyNewF │ │ log-tailer 2026-08-27 094245.737-04:00 INFO (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [BDBEnvironment.setupEnvironmen │ │ log-tailer 2026-08-27 094245.830-04:00 WARN (RepNode kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|77) [StateChangeExecutor.notifyNew │ │ log-tailer 2026-08-27 094246.739-04:00 INFO (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [BDBEnvironment.setupEnvironmen │ │ log-tailer 2026-08-27 094256.767-04:00 ERROR (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [BDBEnvironment.setupEnvironme │ │ log-tailer com.sleepycat.je.rep.InsufficientReplicasException: (JE 18.3.20) Commit policy: SIMPLE_MAJORITY required 1 replica. But none were active with this master. │ │ log-tailer at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureReplicasForCommit(DurabilityQuorum.java:116) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.RepImpl.txnBeginHook(RepImpl.java:1171) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn.txnBeginHook(MasterTxn.java:195) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.initTxn(Txn.java:384) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.<init>(Txn.java:288) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.<init>(Txn.java:267) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn.<init>(MasterTxn.java:146) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn$1.create(MasterTxn.java:117) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn.create(MasterTxn.java:435) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.RepImpl.createRepUserTxn(RepImpl.java:1145) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.createAutoTxn(Txn.java:334) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.LockerFactory.getWritableLocker(LockerFactory.java:79) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.Environment.setupDatabase(Environment.java:816) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.Environment.openDatabase(Environment.java:668) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:292) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:173) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:151) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1211) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1182) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.StarRocksFE.start(StarRocksFE.java:143) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.StarRocksFE.main(StarRocksFE.java:93) ~[starrocks-fe.jar:?] log-tailer 2026-08-27 094301.785-04:00 INFO (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [BDBEnvironment.setupEnvironmen │ │ log-tailer 2026-08-27 094301.790-04:00 WARN (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [StateChangeExecutor.notifyNewF │ │ log-tailer 2026-08-27 094301.793-04:00 INFO (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [BDBEnvironment.setupEnvironmen │ │ log-tailer 2026-08-27 094301.793-04:00 INFO (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [BDBEnvironment.setupEnvironmen │ │ log-tailer 2026-08-27 094311.795-04:00 ERROR (UNKNOWN kube-starrocks-fe-0.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786058030976(-1)|1) [BDBEnvironment.setupEnvironme │ │ log-tailer com.sleepycat.je.rep.InsufficientReplicasException: (JE 18.3.20) Commit policy: SIMPLE_MAJORITY required 1 replica. But none were active with this master. │ │ log-tailer at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureReplicasForCommit(DurabilityQuorum.java:116) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.RepImpl.txnBeginHook(RepImpl.java:1171) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn.txnBeginHook(MasterTxn.java:195) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.initTxn(Txn.java:384) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.<init>(Txn.java:288) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.<init>(Txn.java:267) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn.<init>(MasterTxn.java:146) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn$1.create(MasterTxn.java:117) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.txn.MasterTxn.create(MasterTxn.java:435) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.RepImpl.createRepUserTxn(RepImpl.java:1145) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.Txn.createAutoTxn(Txn.java:334) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.txn.LockerFactory.getWritableLocker(LockerFactory.java:79) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.Environment.setupDatabase(Environment.java:816) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.Environment.openDatabase(Environment.java:668) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:292) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:173) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:151) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1211) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1182) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.StarRocksFE.start(StarRocksFE.java:143) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.StarRocksFE.main(StarRocksFE.java:93) ~[starrocks-fe.jar:?]

This error,

com.sleepycat.je.rep.InsufficientReplicasException

, indicates that the FrontEnd (FE) node acting as the Master cannot find enough healthy "Follower" nodes to reach a quorum (SIMPLE_MAJORITY). In a typical 3-node FE cluster, a quorum requires at least 2 nodes (1 Master + 1 Replica). Your log states

required 1 replica. But none were active with this master

, meaning the Master is running but no other FE nodes have successfully connected to it. This causes the FE process to exit, leading to the reboot cycle you see in Kubernetes. Potential Causes and Solutions 1. Network Connectivity Issues (Most Likely in K8s) The FE pods must be able to communicate with each other using the FQDNs shown in your logs (

kube-starrocks-fe-0.kube-starrocks-fe-search...

). * Action: Ensure all FE pods (

fe-0

,

fe-1

,

fe-2

) are running. Check if they can reach each other on the edit log port (default

9010

). * Action: Verify that the Headless Service (

kube-starrocks-fe-search

) is correctly resolving the pod IPs. 2. Insufficient Active Follower Nodes If you have a 3-node cluster but two FE pods are failing for other reasons (e.g., ImagePullBackOff, OOMKill, or disk pressure), the remaining Master cannot commit any metadata changes and will fail with this exception. * Action: Run

kubectl get pods

to see if the other FE nodes are healthy. If they are also in a crash loop, check their logs for different errors (e.g.,

InsufficientLogException

or

Clock delta

). 3. Clock Desynchronization StarRocks FE nodes require synchronized clocks. If the time difference between nodes exceeds 5 seconds, the BDB JE replication will fail. * Action: Check the

fe.log

on all nodes for a

Clock delta: XXX ms ... exceeds max permissible delta

error. Ensure NTP or a similar service is running on your Kubernetes nodes. 4. Metadata Corruption or Improper Helper Configuration If you recently performed an upgrade or a manual migration, the BDB JE replication group might be in an inconsistent state. * Action: If this is a recovery scenario and you are sure

fe-0

has the latest metadata, you can try starting it with the recovery flag to force it to become the Master without a quorum. Note that this should be a last resort. * Warning: In the FE configuration or start script, ensure the

--helper

points to the correct existing Master node if you are adding new nodes. 5. JVM Memory Pressure The documentation notes that excessive Full GC pauses can cause nodes to be perceived as "down" by the Master, leading to this exception. * Action: Check if the FE pods are hitting memory limits. You may need to increase the JVM heap size (

-Xmx

) in

fe.conf

. Summary of Version Note Note: StarRocks 4.1.3 is not a released version (current stable versions are in the 3.x branch). This may be a typo for 3.1.3 or a custom build. If you are using a very old or development version, consider moving to a stable 3.x release. Recommended Next Step: Check the logs of the other FE pods (

fe-1

and

fe-2

). They likely contain the original error (like a network timeout or disk failure) that prevented them from joining the Master, which in turn caused the Master (

fe-0

) to crash with the

InsufficientReplicasException

. References • administration/Meta_recovery.md • faq/Deploy_faq.md • administration/management/enable_fqdn.md

@Rocky Version was 3.5.20 actually (I was wrong about 4.1.3). Two-node deployment. Here are logs from the second node: │ log-tailer 2026-08-27 094318.396-04:00 ERROR (UNKNOWN kube-starrocks-fe-1.kube-starrocks-fe-search.starrocks.svc.cluster.local_9010_1786462056798(-1)|1) [StarRocksFE.start():207] Star │ │ log-tailer com.starrocks.journal.JournalException: failed to setup environment after retried 3 times │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:307) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:173) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:151) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1211) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1182) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.StarRocksFE.start(StarRocksFE.java:143) ~[starrocks-fe.jar:?] │ │ log-tailer at com.starrocks.StarRocksFE.main(StarRocksFE.java:93) ~[starrocks-fe.jar:?] │ │ log-tailer Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 18.3.20) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid │ │ log-tailer at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:292) ~[starrocks-fe.jar:?] │ │ log-tailer ... 7 more │ │ log-tailer Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.20) kube-starrocks-fe-1.kube-starrocks-fe-search.starrock │ │ log-tailer at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew(ReplicaFeederHandshake.java:432) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:269) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:710) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:486) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:413) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1871) ~[starrocks-bdb-je-18.3.20.jar:?] │ │ log-tailer 2026-08-27 094607.173-04:00 INFO (main|1) [StarRocksFE.start():137] StarRocks FE starting, version: 3.5.20-4d17879 │ │ log-tailer 2026-08-27 094607.185-04:00 INFO (main|1) [FrontendOptions.initAddrUseFqdn():165] Get FQDN from local host by default, FQDN: kube-starrocks-fe-1.kube-starrocks-fe-search.st │ │ log-tailer 2026-08-27 094607.186-04:00 INFO (main|1) [FrontendOptions.initAddrUseFqdn():190] Try to match addr in fqdn mode, ip: 10.42.11.36, FQDN: kube-starrocks-fe-1.kube-starrocks- │ │ log-tailer 2026-08-27 094607.187-04:00 INFO (main|1) [FrontendOptions.initAddrUseFqdn():200] Using FQDN from local host by default, FQDN: kube-starrocks-fe-1.kube-starrocks-fe-search. │ │ log-tailer 2026-08-27 094607.335-04:00 INFO (main|1) [ConsistencyChecker.initWorkTime():133] parsed startDate: 1970-01-01T230000.000-0500, endDate: 1970-01-01T040000.000-0500 │ │ log-tailer 2026-08-27 094607.900-04:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role root[-1] │ │ log-tailer 2026-08-27 094607.937-04:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role db_admin[-2] │ │ log-tailer 2026-08-27 094607.939-04:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role cluster_admin[-3] │ │ log-tailer 2026-08-27 094607.940-04:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role user_admin[-4] │ │ log-tailer 2026-08-27 094607.941-04:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role security_admin[-106] │ │ log-tailer 2026-08-27 094607.942-04:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():343] create built-in role public[-5] │ │ log-tailer 2026-08-27 094608.924-04:00 INFO (main|1) [NodeMgr.getHelperNodes():656] get helper nodes: [kube-starrocks-fe-1.kube-starrocks-fe-search.starrocks.svc.cluster.local:9010] │ │ log-tailer 2026-08-27 094608.939-04:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():508] Current run_mode is shared_data │ │ log-tailer 2026-08-27 094608.939-04:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():513] Got role: FOLLOWER, node name: kube-starrocks-fe-1.kube-starrocks-fe-search.starrocks │ │ log-tailer 2026-08-27 094608.944-04:00 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():340] skip check local environment because helper node and local node are identical. │ │ log-tailer 2026-08-27 094608.996-04:00 INFO (main|1) [BDBEnvironment.setupEnvironment():270] start to setup bdb environment for 1 times

The logs from your second node (

fe-1

) reveal the root cause of the reboot cycle: Clock Desynchronization. The specific error is:

com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew ... exceeds max permissible delta: 5000 ms.

What is happening: 1. FE-1 (Follower) tries to connect to FE-0 (Master). 2. During the handshake, BDB JE (the metadata engine) checks the system time on both nodes. 3. If the time difference (skew) between the nodes is greater than 5 seconds (5000ms), the connection is rejected to prevent metadata inconsistency. 4. Because FE-1 cannot join, FE-0 (the Master) is left alone. In a 2-node deployment, a "simple majority" (quorum) requires both nodes to be active ($2/2 + 1$ is not possible, so it requires all nodes). 5. Since FE-0 cannot reach a quorum, it throws the

InsufficientReplicasException

you saw in your first log and shuts down. How to fix it: 1. Synchronize Clocks (Recommended) You must ensure the system clocks on the underlying Kubernetes worker nodes are synchronized. * Check the time on the nodes running the FE pods:

date -u

. * Install and enable NTP or Chrony on the host machines to keep the time in sync. * Once the host clocks are aligned, restart the FE pods. 2. Increase the Permissible Delta (Workaround) If you cannot immediately sync the clocks, you can temporarily increase the allowed skew in your

fe.conf

. Add or update this parameter:

properties
max_bdbje_clock_delta_ms = 10000

(Setting this to 10000 allows a 10-second difference. Restart the FEs after changing this.) 3. Recommendation on Cluster Size You mentioned this is a two-node deployment. * High Availability: A 2-node FE cluster provides no fault tolerance. If one node fails (or has a clock issue), the whole cluster becomes unwritable because it cannot reach a majority. * Best Practice: Use 3 FE nodes for a production environment. This allows the cluster to remain functional even if one node fails. Summary The crash loop will persist until the time difference between

fe-0

and

fe-1

is less than the

max_bdbje_clock_delta_ms

(default 5s). Synchronizing the host clocks via NTP is the standard fix. References • quick_start/shared-data.md • deployment/shared_data/azure.md • deployment/shared_data/hdfs.md • deployment/shared_data/s3.md