http://starrocks.io logo
Join Slack
Powered by
This message was deleted.
# questions-and-troubleshooting
s
This message was deleted.
r
The metric 
fe_edit_log_size_bytes
represents the disk space consumed by the Berkeley DB Java Edition (BDBJE) edit logs on a StarRocks Frontend (FE) node. These logs record metadata changes (like DDL operations, load job states, and catalog updates). Why the Edit Log Grows It is normal for this metric to increase as metadata changes occur. However, if it grows continuously without decreasing, it usually indicates that old logs are not being reclaimed. Common reasons include: 1. Checkpoint Failure: To reclaim space, the FE must periodically take a "checkpoint" to save its current state into a new 
image
file and discard obsolete edit logs. If the checkpoint process fails (e.g., due to FE memory pressure or OOM), logs will accumulate indefinitely. 2. Lagging Followers: In a High Availability (HA) cluster, the Leader FE retains edit logs so that Follower or Observer nodes can synchronize. If a node is offline or very slow, the Leader may hold onto old logs to help it catch up. 3. High Write Volume: Frequent metadata updates (e.g., thousands of small load jobs or frequent 
REWRITE
operations) generate logs faster than the BDBJE cleaner can reclaim them. 4. BDBJE Cleaner Issues: The internal BDBJE "cleaner" threads responsible for space reclamation may be too few or may be throttled by disk I/O. How to Troubleshoot 1. Check Meta Log Count Monitor the metric 
starrocks_fe_meta_log_count
. This tracks the number of log entries that haven't been checkpointed yet. * Threshold: A value within 100,000 is reasonable. * Action: If it significantly exceeds this (e.g., >500,000), checkpointing is likely failing. 2. Verify Checkpoint Success Search the 
fe.log
of the Leader FE (and the FE performing the checkpoint, usually the leader or a designated follower in newer versions) for checkpoint activities:
Copy code
bash
grep "checkpoint finished" fe.log
If you don't see recent success messages, look for errors like 
OutOfMemoryError
or 
ReplicaWriteException
. Checkpointing is a memory-intensive process; ensure the FE has enough heap memory (
-Xmx
). 3. Check for Lagging Nodes Run the following SQL to check the status of all frontends:
Copy code
sql
SHOW FRONTENDS;
* Ensure all nodes are 
Alive
. * Compare the 
ReplayId
. If one node's 
ReplayId
is significantly lower than the Leader's, it is lagging and preventing log cleanup. 4. Analyze BDBJE Configuration (v3.2+) If the growth is due to slow reclamation rather than checkpoint failure, you can adjust these parameters in 
fe.conf
(requires restart): * `bdbje_cleaner_threads`: Increase the number of background cleaner threads (default is 1). * `bdbje_reserved_disk_size`: Limits the disk space BDBJE reserves for "unprotected" files (default 512MB). 5. Manual Metadata Cleanup (Extreme Cases) If the edit logs have consumed all disk space and the FE cannot start, you may need to perform a metadata recovery. This involves identifying the healthy node with the latest 
image
and 
edit log
, then restarting other nodes using that node as a helper. Refer to the Metadata Recovery guide in the official documentation before attempting this. References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/Meta_recovery.md
When 
starrocks_fe_meta_log_count
drops but 
fe_edit_log_size_bytes
continues to grow, it indicates that the logical checkpointing (StarRocks moving entries into image files) is working, but the physical disk reclamation (BDBJE cleaning up 
.jdb
files) is not keeping pace. This is a common behavior because StarRocks manages metadata logically, while the underlying storage engine (BDBJE) handles physical disk space using its own garbage collection mechanism called the "Cleaner." Why this happens 1. Cleaner Lag: The BDBJE cleaner thread is responsible for identifying 
.jdb
log files that contain mostly "obsolete" data, copying the remaining "active" data to new files, and then deleting the old ones. If the rate of metadata changes is very high, a single cleaner thread may not be able to keep up. 2. Utilization Threshold: BDBJE typically doesn't delete a file the moment a few records are obsoleted. It waits until a file's utilization (percentage of active data) drops below a certain threshold to trigger reclamation. 3. Large Image Files: While the log count drops, the overall metadata size (stored in the image and active BDB logs) might still be increasing due to actual growth in your cluster's metadata (more tables, partitions, or tablets). How to Troubleshoot and Resolve 1. Check for Cleaner activity in logs Search your 
fe.log
(on the Leader FE) for BDBJE cleaner messages. If you see frequent warnings or no "cleaned" messages, the cleaner may be struggling.
Copy code
bash
grep -i "cleaner" fe.log
2. Increase Cleaner Threads (v3.2.0+) If the log growth is due to high write volume, you can increase the number of parallel cleaner threads. * Parameter: 
bdbje_cleaner_threads
(default is 1). * Action: Set this to 
2
or 
4
in 
fe.conf
. * Note: Requires an FE restart to take effect. 3. Adjust Reserved Disk Size (v3.2.0+) StarRocks limits the amount of disk space BDBJE can reserve for "unprotected" (obsolete) files. * Parameter: 
bdbje_reserved_disk_size
(default is 512MB). * Action: If you are on a disk-constrained system, decreasing this value may force BDBJE to free files sooner. However, setting it too low can impact performance. 4. Monitor the physical files Go to your FE metadata directory (defined by 
meta_dir
in 
fe.conf
) and check the 
bdb/
subdirectory:
Copy code
bash
ls -lh fe/meta/bdb/ | head -n 20
If you see a very large number of 
.jdb
files, and many of them are old, the cleaner is likely the bottleneck. 5. Verify Memory Allocation If the FE is under severe memory pressure, the Java-based BDBJE cleaner may run slowly or be frequently interrupted by GC. Ensure your FE has sufficient heap space (
-Xmx
) based on your tablet count (typically 8GB-16GB for medium clusters, 32GB+ for large ones). Summary of Difference Metric | Meaning | Behavior during Checkpoint 
starrocks_fe_meta_log_count
| Logical number of un-checkpointed entries. | Drops immediately after a successful checkpoint. 
fe_edit_log_size_bytes
| Physical disk space of 
.jdb
files. | Only drops when the BDBJE Cleaner deletes a file. References • administration/management/FE_configuration.md • administration/Meta_recovery.md
Open in Slack
PreviousNext
Powered by