@Rocky - shared-data cluster 3.5.16 - three fe nodes, three cn nodes. Deployed through kubernetes operator. It is used mainly to run materialized views against iceberg external catalog. From a clean state it deployed successfully, however, after all materialized views were created now, on restart the fe nodes do not reconnect to cluster and keep crashing. There are plenty of underlying resources and the crashing node is crashing even on the same host where the current leader is successful

Based on the symptoms you described—especially that the FE crashes on restart after creating many Materialized Views (MVs) against an Iceberg catalog—this is likely related to metadata loading exceptions or JVM memory pressure during the journal replay phase. Here are the recommended steps to diagnose and resolve this: 1. Bypass Materialized View Metadata Errors If an MV's metadata becomes inconsistent or encounters an exception during startup (common with complex Iceberg schemas or many partitions), the FE will crash. You can force the FE to ignore these errors to finish starting: * Action: Add the following parameter to your

fe.conf

(or via the Helm chart's

config

section):

properties
    ignore_materialized_view_error = true

* Effect: This allows the FE to skip faulty MV metadata during startup. Once the cluster is up, you can identify and drop/recreate the problematic MVs. 2. Identify the Crash Reason in Logs Check the

fe.log

and

fe.warn.log

on the crashing nodes. Look for specific stack traces: *

RollbackException

or

VLSN

errors: This indicates BDBJE (metadata store) inconsistency between the leader and followers. * `OutOfMemoryError: Java heap space`: Replaying thousands of MV metadata entries can spike memory usage. Ensure the FE

Xmx

is sufficient (typically 8GB+ for production). * `failed to load journal type...`: Indicates a corrupted journal entry. 3. Handle Metadata Inconsistency (Follower Reconnection) Since the leader is healthy but followers cannot reconnect, the followers' local metadata might be out of sync. In a Kubernetes environment, you can force a clean sync: 1. Stop the crashing follower FE pods. 2. Clear the metadata directory (the

meta/

folder) on the persistent volume of the crashing follower nodes. 3. Restart the pods. They should rejoin as "clean" nodes and pull a fresh image from the Leader FE. _ _Note: Ensure the

edit_log_port

(default 9010) and

cloud_native_meta_port

(default 6090) are reachable between nodes.* 4. Shared-Data Specific Configuration In version 3.5.x, StarRocks requires JDK 17. Verify that your deployment is not using an older JDK or incompatible

JAVA_OPTS

from a previous version (like CMS GC flags which are deprecated). Additionally, for clusters with heavy external catalog usage, add these to

fe.conf

to reduce resource consumption during metadata processing:

properties
proc_profile_mem_enable = false
proc_profile_cpu_enable = false

5. Check for Journal ID Corruption If the logs show a specific "Bad Journal ID," you may need to skip it:

properties
metadata_journal_skip_bad_journal_ids = <erroneous_id>

Caution: Skipping journals can lead to minor metadata loss (like the last few DDLs/MVs created). Summary of Recommended

fe.conf

for Recovery:

properties
# Bypasses MV-related startup crashes
ignore_materialized_view_error = true

# If replaying is extremely slow/stuck
# metadata_enable_recovery_mode = true (Use only as a last resort)

If the FE continues to crash with

ignore_materialized_view_error = true

, please check the last 50 lines of

fe.log

for a

CAUSED BY

clause and share that specifically. References • administration/Meta_recovery.md • administration/management/FE_configuration.md

@Rocky - any drawbacks on keeping

ignore_materialized_view_error=true

While

ignore_materialized_view_error = true

is an effective "emergency bypass" to get your FE nodes back online, keeping it enabled long-term is not recommended. Here are the specific drawbacks and risks of keeping this parameter active: 1. Persistent Metadata Inconsistency This flag does not "fix" the underlying metadata error; it simply tells the FE to skip the corrupted or problematic journal entry during the startup replay. * The Result: The specific Materialized View (MV) that caused the crash will likely be in an inconsistent or "broken" state. It may appear in

SHOW MATERIALIZED VIEWS

but fail to refresh, fail to rewrite queries, or return incorrect results. 2. Silent Failures If you leave the flag on, future metadata errors related to MVs will also be silently ignored. * The Risk: Instead of a clear signal (a crash) that your metadata is becoming corrupted—perhaps due to a bug in how Iceberg partitions are tracked—the FE will continue running with "ghost" or broken metadata. This makes troubleshooting much harder later on because you won't know exactly when or why a metadata entry failed to load. 3. Potential for "Dead" Objects If the FE skips a journal entry that was supposed to complete a

CREATE

or

ALTER

operation for an MV, you may end up with "zombie" metadata objects that consume memory or prevent you from creating a new MV with the same name. 4. Difficulty in Identifying the Root Cause By bypassing the error, you lose the immediate stack trace that identifies the specific problematic MV. This is particularly risky when using Iceberg catalogs, where schema evolutions or partition changes in the external source can sometimes trigger edge cases in StarRocks' metadata management. ────────── Recommended Workflow (The "Clean" Way) Rather than keeping the flag on indefinitely, use it as a recovery step: 1. Recover: Start the FEs with

ignore_materialized_view_error = true

. 2. Identify: Once the cluster is up, check the status of your MVs:

sql
    SELECT * FROM information_schema.materialized_views WHERE is_active = 'false';
    -- or
    SHOW MATERIALIZED VIEWS;

3. Cleanup: Manually

DROP

any MVs that are inactive or behaving strangely. 4. Create Image: Once the cluster is healthy and the "bad" MVs are removed, force a new metadata checkpoint:

sql
    ALTER SYSTEM CREATE IMAGE;

5. Revert: Remove

ignore_materialized_view_error = true

from your

fe.conf

(or Helm config) and restart the FEs one by one to ensure they can now start cleanly under normal conditions. 6. Recreate: Re-create the MVs you dropped. Summary: Treat this flag like a "spare tire"—it's great for getting you to the garage, but you shouldn't drive on it forever. References • administration/management/FE_configuration.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • sql-reference/sql-statements/materialized_view/DROP_MATERIALIZED_VIEW.md

@Rocky - before this setting fe1 and fe2 were up, fe0 was crashing. After this setting picture is still the same

If

fe1

and

fe2

are running but

fe0

continues to crash despite the

ignore_materialized_view_error

setting, the issue is likely local metadata corruption or a BDBJE replication conflict on

fe0

. Because

fe1

and

fe2

are healthy, the cluster’s "source of truth" is safe. You can fix

fe0

by forcing it to perform a clean synchronization from the current leader. 1. Identify the Exact Crash Reason Check the logs on

fe0

for these specific signatures: *

RollbackException

or

VLSN

mismatch: This happens if

fe0

has written local metadata that conflicts with the leader's history. * `java.io.IOException: Insufficient space`: Even if the host has space, the PVC assigned to

fe0

might be full. * `com.sleepycat.je.EnvironmentFailureException`: Indicates a corruption in the local BDBJE environment files. 2. The "Clean Rejoin" Procedure (Recommended) Since this is a Kubernetes deployment and you have a working leader, the most reliable fix is to wipe the metadata on the crashing node and let it re-sync. 1. Stop the

fe0

pod: If the operator is auto-restarting it, you may need to temporarily scale the FE replicas down to 2, or use

kubectl delete pod

and immediately perform the next step. 2. Clear the metadata directory: You need to delete the contents of the

meta/

directory for

fe0

. * Find the PVC associated with

kube-starrocks-fe-0

. _ You can use a temporary "debugger" pod to mount that PVC and run

rm -rf /opt/starrocks/fe/meta/_

. 3. Restart `fe0`: The StarRocks Operator will restart the pod. It should recognize its meta directory is empty and perform a "helper" start, pulling a fresh image from the current Leader (

fe1

or

fe2

). 3. Check Kubernetes Resource Limits Creating many Materialized Views significantly increases the JVM Heap usage during the "Image Loading" phase of startup. * Memory: If

fe0

is hitting its K8s

limit

, it will be

OOMKilled

before it can finish replaying the journals. Compare the memory usage of the running

fe1

pod; if it is near the limit, you must increase the memory requests/limits in your

StarRocksCluster

YAML. * Liveness Probe: If the metadata is very large (common with Iceberg MVs), the FE might take a long time to start. Check if the K8s Liveness Probe is killing the pod before it finishes loading. Increase

initialDelaySeconds

if necessary. 4. Shared-Data Port & Role Conflicts In StarRocks 3.5 Shared-Data mode, ensure that

fe0

can communicate with the leader on these ports: * 9010 (Edit Log Port) * 6090 (Cloud Native Metadata Port) If

fe0

was the original leader and the cluster was redeployed, there might be a mismatch in the

meta/image/ROLE

file or

meta/image/VERSION

. Clearing the metadata (as described in step 2) resolves this. 5. Java/JDK 17 Check StarRocks 3.5.x requires JDK 17. If your K8s environment or custom

fe.conf

still contains legacy Java 8 parameters (like

-XX:+UseConcMarkSweepGC

), the JVM will fail to start on newer images. Ensure your

JAVA_OPTS

are clean and compatible with JDK 17. Next Step: If the pod still crashes after wiping the meta directory, please provide the output of

kubectl logs <fe0-pod-name> --tail=100

. References • administration/Meta_recovery.md • faq/Deploy_faq.md • deployment/shared_data/azure.md

@Rocky - JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx2048m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time" Memory wise Fe1:

Memory: 75.6 GiB/125.4 GiB
Swap used: 388.8 MiB/8.0 GiB

FE2:

Memory: 89.6 GiB/125.4 GiB
Swap used: 60.5 MiB/8.0 GiB

Liveness settings are on default: # StartupProbeFailureSeconds defines the total failure seconds of a startup Probe. # the default value is 300 seconds # You can set it to "0" to disable the probe. startupProbeFailureSeconds: # LivenessProbeFailureSeconds defines the total failure seconds of a liveness Probe. # the default value is 15 seconds # You can set it to "0" to disable the probe. livenessProbeFailureSeconds: # ReadinessProbeFailureSeconds defines the total failure seconds of a readiness Probe. # the default value is 15 seconds # You can set it to "0" to disable the probe. readinessProbeFailureSeconds:

The cause of the crash on

fe0

is your JVM Heap size. Setting

-Xmx2048m

(2GB) for a StarRocks FE—especially one managing Materialized Views against an Iceberg catalog—is significantly below the recommended minimum. StarRocks stores all metadata in memory. Replaying the journal logs for multiple MVs (which includes partition info, schema metadata, and file lists from Iceberg) will almost certainly exceed 2GB, causing the JVM to crash with an

OutOfMemoryError

or become unresponsive during startup. 1. Increase FE Memory (Crucial) StarRocks official documentation recommends a minimum of 16 GB for the FE heap even for small clusters. Given your hosts have 125 GiB of RAM, you should increase this immediately. Action: Update your

JAVA_OPTS

in the

StarRocksCluster

YAML or `fe.conf`:

bash
# Recommended for your host size (matching Xms and Xmx is best practice)
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xms16384m -Xmx16384m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"

Note: If you have thousands of tablets or complex Iceberg MVs, consider 32GB. 2. Adjust Kubernetes Probes Your

livenessProbe

is set to fail after 15 seconds. When an FE starts, it must: 1. Load the metadata image from disk. 2. Replay all journal logs since the last image. 3. Synchronize with the Leader. With 2GB of RAM, the JVM is likely spending all its time in "Full GC," making it unable to respond to the K8s health check within 15 seconds. Even after increasing memory, loading large metadata can take longer than 15s. Action: Increase the

livenessProbe

and

readinessProbe

grace periods in your YAML:

yaml
livenessProbeFailureSeconds: 300  # Give it 5 minutes to recover/sync
readinessProbeFailureSeconds: 300

3. Clear

fe0

Metadata (If it still won't join) Since

fe1

and

fe2

are healthy, they have the correct "Source of Truth." If

fe0

continues to crash after the memory increase, its local BDBJE metadata might be in an inconsistent state from the previous crashes. Action: 1. Scale the FE replicas down or stop the

fe0

pod. 2. Wipe the

/opt/starrocks/fe/meta

directory on the Persistent Volume (PVC) associated with

fe0

. 3. Restart the pod. It will rejoin as a blank node and pull a clean, healthy metadata image from the Leader (

fe1

or

fe2

). Summary of Recommendations: * Primary Fix: Increase FE Heap (

-Xmx

) to at least 16GB. * Stability Fix: Increase Kubernetes Probe failure thresholds to at least 300s. * Cleanup: Keep

ignore_materialized_view_error = true

only until

fe0

successfully joins, then remove it and perform an

ALTER SYSTEM CREATE IMAGE;

to consolidate the metadata. References • faq/Others.md (FE Memory Table) • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_memory_problems.md References • knowledge/trouble_shooting/fe_memory_problems.md • faq/Others.md • administration/Meta_recovery.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md

Hello, @Aleksandr Zakharchenko! How can I assist you today?

@Rocky 2026-05-12 104730.882-04:00 INFO (UNKNOWN kube-sr-fe-0.kube-sr-fe-search.sr.svc.mycorp.com_9010_1778537364201(-1) |1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: FOLLOWER. is ready: false 2026-05-12 104730.883-04:00 WARN (UNKNOWN kube-sr-fe-0.kube-sr-fe-search.sr.svc.mycorp.com_9010_1778537364201(-1) |1) [GlobalStateMgr.waitForReady():1232] It took too much time for FE to transfer to a stable state(LEADER/FOLLOWER), it maybe caused by one of the following reasons: 1. There are too many... 2026-05-12 104732.883-04:00 INFO (UNKNOWN kube-sr-fe-0.kube-sr-fe-search.sr.svc.mycorp.com_9010_1778537364201(-1) |1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: FOLLOWER. is ready: false 2026-05-12 104733.604-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84040 to 84041, count 1, cost 9ms. 2026-05-12 104733.657-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84041 to 84048, count 7, cost 48ms. 2026-05-12 104734.884-04:00 INFO (UNKNOWN kube-sr-fe-0.kube-sr-fe-search.sr.svc.mycorp.com_9010_1778537364201(-1) |1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: FOLLOWER. is ready: false 2026-05-12 104736.588-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84048 to 84049, count 1, cost 7ms. 2026-05-12 104736.604-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84049 to 84050, count 1, cost 13ms. 2026-05-12 104736.628-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84050 to 84051, count 1, cost 18ms. 2026-05-12 104736.653-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84051 to 84056, count 5, cost 7ms. 2026-05-12 104736.685-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84056 to 84057, count 1, cost 19ms. 2026-05-12 104736.709-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84057 to 84058, count 1, cost 4ms. 2026-05-12 104736.731-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84058 to 84059, count 1, cost 5ms. 2026-05-12 104736.752-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84059 to 84064, count 5, cost 19ms. 2026-05-12 104736.885-04:00 INFO (UNKNOWN kube-sr-fe-0.kube-sr-fe-search.sr.svc.mycorp.com_9010_1778537364201(-1) |1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: FOLLOWER. is ready: false 2026-05-12 104738.886-04:00 INFO (UNKNOWN kube-sr-fe-0.kube-sr-fe-search.sr.svc.mycorp.com_9010_1778537364201(-1) |1) [GlobalStateMgr.waitForReady():1228] wait globalStateMgr to be ready. FE type: FOLLOWER. is ready: false 2026-05-12 104740.674-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84064 to 84065, count 1, cost 7ms. 2026-05-12 104740.692-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84065 to 84066, count 1, cost 13ms. 2026-05-12 104740.705-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84066 to 84067, count 1, cost 5ms. 2026-05-12 104740.726-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84067 to 84071, count 4, cost 3ms. 2026-05-12 104740.738-04:00 INFO (star mgr replayer|133) [StarOSDGBJEJournalSystem.replayJournal():243] star mgr replayed journal from 84071 to 84072, count 1, cost 5ms.

The logs confirm that

fe0

is actually "healthy" in terms of connectivity (it knows it is a

FOLLOWER

and is talking to the leader), but it is struggling to catch up with the metadata. Here is the breakdown of why it is still failing and how to fix it: 1. The "GC Death Spiral" (Heap vs. Reality) Your logs show the FE replaying journals (e.g.,

84040 to 84072

). This is a very slow rate (a few journals per second). * The Conflict: You have restricted the JVM to 2GB (

-Xmx2048m

), but your other FE nodes (

fe1

,

fe2

) are currently using 75GB - 89GB of memory. * The Result:

fe0

is trying to load 75GB+ worth of metadata into a 2GB container. It is likely spending 99% of its CPU time on Garbage Collection (GC) trying to free up space that doesn't exist. This makes the journal replay crawl at a snail's pace. Action: You must increase the FE memory. For a cluster where the metadata already occupies 75GB+, you should set your heap to at least 96GB or 100GB to match the existing nodes.

bash
# Example change for your StarRocksCluster YAML
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xms96g -Xmx96g -XX:+UseG1GC"

2. Kubernetes Probes are killing the process Your

livenessProbe

and

readinessProbe

are set to fail after 15 seconds. * Because the FE is replaying journals (which can take minutes or even hours if there are many updates), it will never become "ready" within 15 seconds. * Kubernetes sees the FE is "not ready" for 15 seconds and kills the pod, starting the whole slow process over again. This is why

fe0

stays in a crash/restart loop while the others are up. Action: Increase your probe failure thresholds significantly to allow the FE time to load metadata.

yaml
startupProbeFailureSeconds: 3600  # Give it up to 1 hour to load the image/journals
livenessProbeFailureSeconds: 600   # 10 minutes
readinessProbeFailureSeconds: 600

3. Why

ignore_materialized_view_error

didn't "fix" it That setting prevents a crash if a specific corrupted entry is found in the logs. However, it doesn't speed up the replay process or fix the lack of memory.

fe0

is simply doing the work of catching up, but it's doing it with no "breathing room" (RAM). 4. How to fix

fe0

quickly (The "Fast Sync" Method) Instead of waiting for

fe0

to replay tens of thousands of journals at 2GB RAM: 1. Increase the RAM as noted in Step 1. 2. Delete the metadata directory on `fe0`'s PVC:

bash
    # Path inside the container is usually:
    rm -rf /opt/starrocks/fe/meta/*

3. Restart `fe0`: Because the directory is empty,

fe0

will perform a "Helper Start." It will download a completed image from the Leader (

fe1/fe2

) instead of replaying individual journals. Downloading a 75GB image is significantly faster than replaying the journal history that created it. Summary * Immediate Fix: Set

-Xmx

to at least 96g and increase

startupProbeFailureSeconds

to 3600. * Verification: Run

SHOW PROC '/frontends';

on

fe1

. Check the

ReplayedJournalId

of

fe1

vs

fe0

. If

fe0

is far behind (e.g.,

fe1

is at 1,000,000 and

fe0

is at 84,000), it will take a long time to catch up unless you wipe the meta and let it sync the image. References • administration/Meta_recovery.md • faq/Deploy_faq.md • deployment/deploy_manually.md

@Rocky - apologies dev config was shown for JAVA_OPTS. In Prod allocation is JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx12288m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time" It is the same for all nodes - not sure where the web ui pulls down the memory allocation from, as limit is set to 30GB per node

The memory discrepancy you see in the Web UI is a common point of confusion. Here is the explanation for your current situation and why

fe0

is still failing: 1. Discrepancy: Web UI vs.

JAVA_OPTS

The memory reported in the StarRocks Web UI (e.g.,

75.6 GiB/125.4 GiB

) comes from the host hardware via the OSHI library (controlled by

http_web_page_display_hardware

), not from the JVM heap. * 125.4 GiB: This is the total physical RAM on the underlying worker node. * 75.6 GiB / 89.6 GiB: This is the total memory used by all processes on that host (including other pods/services), not just the FE process. * Your JVM Heap: Your

JAVA_OPTS

limits the FE's internal memory to 12GB (

-Xmx12288m

). Even though the host has 125GB, the StarRocks process is restricted to only using 12GB of it. 2. Why

fe0

is still crashing Even with 12GB,

fe0

is likely stuck in a "Liveness Probe Death Spiral": * Slow Journal Replay: Your logs show

fe0

is replaying journals (

84040 to 84041

, etc.). This process takes time, especially if the metadata image is large. * Probe Timeout: Your

livenessProbe

and

readinessProbe

are set to fail after 15 seconds. * The Conflict: If

fe0

takes more than 15 seconds to finish replaying its journals and become "Ready," Kubernetes will kill the pod and restart it from scratch. It never gets enough time to finish the job. 3. Recommended Actions A. Increase Probes (Immediate Priority) Give the FE enough time to start up and sync metadata without being killed.

yaml
# In your StarRocksCluster YAML
startupProbeFailureSeconds: 3600  # Allow 1 hour for first startup
livenessProbeFailureSeconds: 600   # 10 minutes for health check
readinessProbeFailureSeconds: 600

B. Increase JVM Heap For a production environment, 12GB is below the recommended minimum. StarRocks recommends at least 16GB for clusters with fewer than 1 million tablets. If you are using Iceberg Materialized Views, you should allocate more, as Iceberg metadata is cached in the FE. Since your container limit is 30GB, you have room to increase the heap:

bash
# Recommended: Increase to 24GB to leave 6GB for overhead/off-heap
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xms24576m -Xmx24576m -XX:+UseG1GC ..."

C. Perform a "Helper Start" (Fastest Fix) If

fe0

is many thousands of journals behind, replaying them one-by-one is inefficient. You can force it to pull a fresh metadata image from the leader: 1. Stop

fe0

(or let it crash). 2. Wipe the metadata directory on the PVC for `fe0`:

rm -rf /opt/starrocks/fe/meta/*

. 3. Restart

fe0

. 4. Because the meta folder is empty, it will perform a "Helper Start," download the full healthy image from

fe1

or

fe2

, and join the cluster much faster. How to verify actual JVM memory To see how much of your 12GB is actually being used by the JVM (rather than the host), run this inside the

fe1

or

fe2

pod:

bash
jstat -gcutil $(cat /opt/starrocks/fe/bin/fe.pid) 1000 5

Check the O (Old Generation) column. If it is consistently >90%, your 12GB heap is full, and you must increase it. References • faq/Others.md • knowledge/trouble_shooting/fe_memory_problems.md • data_source/catalog/iceberg/iceberg_catalog.md • administration/management/FE_configuration.md

@Rocky - adding memory to 16GB and increasing timeout on startup helped - it required 10 minutes in my case to come to sync.

That is great news. The 10-minute sync time confirms that the FE was busy replaying a large volume of metadata journals or loading a large image file into memory—both of which were being interrupted by the previous 15-second timeout. To ensure the cluster stays healthy and avoids long startup times in the future, you may want to check two things: 1. Check the Checkpoint Status If startup takes 10 minutes, it often means the "Checkpoint" process (which merges journal logs into a single image file) hasn't happened recently, or the image itself has grown quite large. Run this in your SQL client:

sql
SHOW PROC '/frontends';

* LastCheckpointTime: Ensure this is recent (usually within the last hour). If it's days old, the FE Leader might be struggling to create checkpoints. * ReplayedJournalId: Compare this across all three nodes. They should be very close to each other. 2. Monitor the Heap Usage Now that you are at 16GB, monitor the "Old Gen" memory usage. If you add more Materialized Views or the Iceberg catalog grows, 16GB may become tight again. * Tip: If you see the FE becoming sluggish or sync times increasing further, consider moving to 24GB or 32GB, as your host hardware (125GB) has plenty of room to support it. Why the 10 minutes was necessary: During that time, the FE was performing two heavy tasks: 1. Image Loading: Loading the

image.<id>

file from disk into the JVM Heap. 2. Journal Replay: Replaying every transaction (create table, insert, MV refresh) that happened since the last image was saved. With the timeout now set higher, the FE can finally finish this work without Kubernetes "pulling the plug" midway through-plug" halfway through. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • knowledge/trouble_shooting/fe_memory_problems.md )