@Rocky How can we reduce FE pod startup time we have currently 120 Dbs and with 11 cpus and 110Gi mem and heap memory around 85Gi. It is taking around 1 hr just to start when failed. any way we can reduce it ? Most of the time it consumes for to catch up the journal

To reduce StarRocks FE startup time when it is delayed by journal (edit log) catch-up, the primary goal is to ensure that metadata checkpoints (images) are created frequently and successfully. When an FE starts, it loads the latest

image

file and سپس replays all edit logs generated after that image was created. If an image hasn't been created recently, the FE must replay thousands (or millions) of journal entries, which is a single-threaded process and often the cause of 1-hour startup times. 1. Optimize Checkpoint Frequency The FE creates a new

image

file once the number of edit log entries exceeds a certain threshold. *

edit_log_roll_num

: This parameter (default

50000

) controls how many edit log entries are written before a log file is rolled. It also influences how often the

CheckpointController

triggers an image creation. If your system has high metadata churn (e.g., many routine loads, frequent partitions), consider decreasing this value (e.g., to

20000

) to force more frequent checkpoints. * Verify Checkpoint Status: Check your FE

meta/image/

directory. If the timestamp of the latest

image.<id>

file is very old (e.g., several hours ago), checkpoints are likely failing. Search your

fe.log

for keywords like

checkpoint

or

image

to see if there are errors during image creation. 2. FE Configuration Tuning *

checkpoint_only_on_leader

: (Available in v3.4+) By default, StarRocks may pick a non-leader FE to perform the checkpoint to save leader resources. If you have a multi-FE setup and checkpoints are failing or slow, try setting this to

true

to ensure the Leader (which has the most up-to-date state) handles the image creation. *

bdbje_cleaner_threads

: (Available in v3.2+) If your journal is stored in BDBJE, increasing the number of cleaner threads (default

1

) can help reclaim disk space faster and reduce I/O contention during replay. Try setting this to

4

or

8

. *

metadata_journal_max_batch_cnt

: Increasing the batch size of journal writes can improve efficiency, but for replay specifically, the bottleneck is often the processing of each entry. 3. Check for External Blocking (v3.x / v4.x) In newer versions, if you use External Catalogs (Hive, Iceberg), journal replay can sometimes block on external metadata lookups (like HMS or HDFS) for statistics. * Check if you have many external tables. A known bug (fixed in recent patches) can cause the replayer to stall if the external metastore is slow. * If your FE logs show it's stuck on

AnalyzeMgr.refreshConnectorTableBasicStatisticsCache

during replay, this is likely the cause. 4. Infrastructure & Resource Allocation * Storage Speed: The

meta_dir

should ideally be on SSD/NVMe storage. Journal replay is I/O intensive; slow disk latency directly translates to slow startup. * JVM Heap: Your heap is 85Gi, which is plenty for 120 DBs. However, ensure that you are not hitting swap. With 110Gi total memory and 85Gi heap, you have enough overhead, but monitor

fe.gc.log

to ensure "Stop-the-World" GC isn't happening during the startup phase. * CPU: Metadata replay is mostly single-threaded. While you have 11 CPUs, only one core will be heavily utilized during the catch-up phase. High-frequency CPUs are better than many-core CPUs for this specific task. 5. Manual Intervention (Last Resort) If the FE is stuck in a loop or replaying an impossible amount of journals: * You can manually trigger a checkpoint on a running Leader FE using

ALTER SYSTEM CHECKPOINT;

(if available in your version) or by simply letting the cluster run until it naturally checkpoints. * Ensure that the

meta_dir

has enough free space (at least 10GB recommended); if the disk is full, checkpoints will fail silently, leading to an ever-growing journal. Recommendation:

we are on 4.0.14 version @Rocky

In StarRocks v4.0.14, a 1-hour startup time for 120 databases is typically caused by a failure to create metadata checkpoints (images) frequently, or the replayer thread getting blocked by external metadata lookups. Here is how you can reduce the FE startup and journal catch-up time: 1. Identify the Bottleneck: External Statistics In v4.0.x (specifically noted around v4.0.10+), a known issue (Bug #73832) can cause the FE journal replayer to block while replaying statistics for External Catalogs (Hive, Iceberg). * The Problem: When the replayer encounters a journal entry for external table statistics, it may attempt a synchronous call to the external metastore (HMS/HDFS). If that metastore is slow or the network is latent, startup hangs. * The Fix: Check if your

fe.log

shows the replayer thread stuck on

refreshConnectorTableBasicStatisticsCache

. If you have many external tables, consider upgrading to v4.1+ which introduces a config to perform this refresh lazily (after startup). For v4.0.14, ensure your external metastores are highly available during FE startup. 2. Force More Frequent Checkpoints If your FE is replaying a massive number of journals, it means the last "image" (snapshot) is too old. *

edit_log_roll_num

: Default is

50000

. If you have high metadata churn (many loads/partitions), the journal grows fast. Reduce this to

20000

to force the FE to create images more frequently. * Check the Image File: Look in your

meta/image/

directory. If the timestamp of the latest

image.<id>

file is more than an hour old on the Leader FE, checkpoints are failing. Search

fe.log

for "checkpoint" errors (e.g., Disk Full, JVM OOM). 3. Optimize Checkpoint Execution *

checkpoint_only_on_leader

: Default is

false

. By default, StarRocks tries to run checkpoints on non-leader FEs to save leader resources. However, if your followers have resource contention or slower disks, they may fail to create images, causing the journal to never be truncated. Set this to

true

to ensure the Leader (which has the most consistent state) handles the checkpointing. *

bdbje_cleaner_threads

: Default is

1

. For large metadata (85Gi heap), the internal Berkeley DB (BDBJE) needs to clean old log files aggressively. Increase this to

4

or

8

in

fe.conf

. 4. Hardware and JVM Considerations * Storage Speed: Ensure the

meta_dir

is on SSD or NVMe. Journal replay is a single-threaded I/O-intensive operation. 1 hour of catch-up on 11 CPUs suggests the replayer is likely "I/O bound" or "latency bound" rather than CPU bound. * JVM Replay Speed: Monitor your

fe.gc.log

during startup. With an 85Gi heap, if the replayer is creating millions of small objects during catch-up, you might be hitting long GC pauses. Ensure your

-Xms

and

-Xmx

are both set to

85G

to avoid heap resizing during startup. 5. Deployment Policy (v3.2+ / v4.0+) If you have 3 or more Follower FEs, change these policies in

fe.conf

to reduce the I/O overhead of writing/replaying logs: *

master_sync_policy = WRITE_NO_SYNC

*

replica_sync_policy = WRITE_NO_SYNC

(Note: These settings improve performance but slightly increase the risk of losing the last few milliseconds of metadata in a total power-loss scenario.) Summary Checklist: 1. Check if

meta/image

has a recent file. If not, checkpoints are failing. 2. Set

checkpoint_only_on_leader = true

and

edit_log_roll_num = 20000

. 3. Increase

bdbje_cleaner_threads = 8

. 4. Check for external catalog connectivity issues blocking the replayer. References • administration/management/FE_configuration.md • faq/Deploy_faq.md

[ec2-user@ ~]$ ksr logs reporting-dedicated-3-fe-0 --tail=-1 | grep "refreshConnectorTableBasicStatisticsCache" [ec2-user@ ~]$ [ec2-user@~]$ @Rocky

Since your grep for

refreshConnectorTableBasicStatisticsCache

returned no results, the issue is likely not the specific external catalog statistics bug. Instead, the 1-hour startup time is likely caused by massive journal accumulation due to infrequent or failing metadata checkpoints (images). With 120 DBs and an 85Gi heap, your metadata is substantial. If the system has to replay millions of edit logs upon startup, it will be extremely slow. 1. Verify Checkpoint Health If your FE takes an hour to catch up, it means the "image" file is likely very old. * Check the Log: Search your Leader FE's

fe.log

for these specific phrases to see if checkpoints are even finishing:

bash
    grep -E "begin to generate new image|checkpoint finished save image" fe.log

* Check the Disk: Look at the

meta/image

directory. If the latest

image.<id>

file has a timestamp from several hours (or days) ago, the checkpoint process is failing or not triggering often enough. 2. Tuning for Faster/More Frequent Checkpoints To reduce startup time, you need to ensure the image is updated more frequently so there are fewer logs to replay: *

edit_log_roll_num

: Reduce this from the default

50000

to

20000

. This forces the FE to create a new image file sooner. *

checkpoint_only_on_leader = true

: In v4.0.14, ensuring the Leader FE handles the checkpointing can prevent failures caused by "Follower lag" or lower-resourced followers attempting the checkpoint. *

bdbje_cleaner_threads = 8

: This increases the background threads for cleaning up old Berkeley DB log files, which reduces disk fragmentation and I/O overhead during both operation and replay. 3. Check for Disk I/O Bottlenecks Journal replay is a single-threaded, I/O-intensive operation. * Latency: Ensure your

meta_dir

is on high-performance storage (SSD/NVMe). If you are using network-attached storage (like EBS gp2/gp3) with low IOPS, the replay will crawl. * Log Errors: Search for

LogbufferPool_FullLatch

or

Latch timeout

in

fe.log

. If you see these, your disk cannot keep up with the metadata writes/reads, causing the startup to stall. 4. Kubernetes Probes (Startup Issues) If you are in K8s, a 1-hour startup might trigger a

CrashLoopBackOff

if your Readiness or Liveness probes are too aggressive. * Ensure your

initialDelaySeconds

or

failureThreshold

for the probes are high enough to allow the FE to finish catching up without being killed and restarted by K8s (which restarts the 1-hour process from scratch). 5. Check "Current Replay" Progress During the startup, you can see exactly which journal entry the FE is on. Run this on the starting FE node:

bash
grep "replayed to vlsn" fe.log | tail -n 5

Compare the

vlsn

(Version Log Sequence Number) to the one on the current Leader. If the number is increasing very slowly, the bottleneck is I/O or a specific expensive metadata operation being replayed (like a massive drop/add partition). Recommendation: Start by setting

edit_log_roll_num = 20000

and

checkpoint_only_on_leader = true

. Then, monitor the

meta/image

folder to ensure new images are created every 15-30 minutes during normal operation. This will ensure that if a pod fails, it only has a few minutes of logs to replay. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md