http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> how does the LOG_LEVEL variable inf...
# questions-and-troubleshooting
f
@Rocky how does the LOG_LEVEL variable influence the log files? do the contents change or do the files like be.INFO stop appearing if log level is error?
r
The 
sys_log_level
variable (often referred to as 
LOG_LEVEL
in some contexts) controls the minimum severity level of messages that are recorded. Setting it to 
ERROR
changes the content of the log files, but the files themselves do not stop appearing. In StarRocks, the Backend (BE) logging is based on Google glog, which follows specific rules for how logs are distributed across files: 1. Log Filtering (Content Changes) When you set 
sys_log_level
to 
ERROR
, the system ignores any log entries with a severity lower than 
ERROR
. * INFO and WARNING logs will no longer be generated or written anywhere. * Only ERROR and FATAL logs will be processed. 2. Log File Distribution (Why files still appear) The 
<http://be.INFO|be.INFO>
, 
be.WARNING
, and 
be.ERROR
files are organized by "minimum level." In glog's default behavior (which StarRocks uses): * 
<http://be.INFO|be.INFO>
: Records everything at 
INFO
level and higher (includes INFO, WARN, ERROR, FATAL). * 
be.WARNING
: Records everything at 
WARN
level and higher (includes WARN, ERROR, FATAL). * 
be.ERROR
: Records everything at 
ERROR
level and higher (includes ERROR, FATAL). If the log level is set to 
ERROR
, any error that occurs is written to all three files. Therefore, even if you set the level to 
ERROR
, the 
<http://be.INFO|be.INFO>
file will still exist and be updated, but it will only contain messages that are 
ERROR
or 
FATAL
. It acts as a "catch-all" for everything the system is currently allowed to log. Summary Table 
sys_log_level
 | 
<http://be.INFO|be.INFO>
contains | 
be.WARNING
contains | 
be.ERROR
contains 
INFO
 (Default) | INFO, WARN, ERROR, FATAL | WARN, ERROR, FATAL | ERROR, FATAL 
WARN
 | WARN, ERROR, FATAL | WARN, ERROR, FATAL | ERROR, FATAL 
ERROR
 | ERROR, FATAL | ERROR, FATAL | ERROR, FATAL Note: For the Frontend (FE), which uses log4j2, 
fe.log
behaves similarly to 
<http://be.INFO|be.INFO>
(recording everything allowed by the current level), and 
fe.warn.log
behaves like 
be.WARNING
. Setting the level to 
ERROR
will similarly restrict their content to only Errors/Fatals without deleting the files. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/logs.md
f
@Rocky 
<http://be.INFO|be.INFO>
: Records everything at 
INFO
level and higher (includes INFO, WARN, ERROR, FATAL). is that explained in the docs somewhere?
r
The documentation for StarRocks explicitly mentions the "and above" behavior for the Warning and Error logs, though it uses slightly different phrasing for the Info log. Here is how it is documented in the BE/CN Logging in detail section: * 
be.WARN.log
: "stores log entries at WARNING level and above." * 
be.ERROR.log
: "stores log entries at ERROR level and above." * 
be.INFO.log
: Described as primarily recording logs "at the INFO level." While the BE section doesn't use the exact phrase "and above" for the INFO log, the FE Logging section (which follows a similar logic) explicitly clarifies this relationship:
"Compared to 
fe.log
, which records logs of all levels, 
fe.warn.log
focuses on warnings and errors..."
Why this happens The StarRocks Backend (BE) uses Google glog. According to the standard glog behavior (which StarRocks implements):
"A log message is written to the log file corresponding to its severity and to all log files of lower severity. For example, a message of severity 
ERROR
will be logged to the 
ERROR
, 
WARNING
, and 
INFO
logs."
Summary of what's in the docs: 1. Level Inheritance: The docs explicitly state 
WARN
and 
ERROR
files include higher levels. 2. All Levels: The 
fe.log
(FE equivalent of 
<http://be.INFO|be.INFO>
) is explicitly described as "all levels." 3. Sys Log Directory: The 
sys_log_dir
parameter is described as the directory for "system logs (including INFO, WARNING, ERROR, and FATAL)," treating them as a nested set. If you set 
sys_log_level = ERROR
, the "minimum severity" filter is applied first. Since INFO and WARN messages are no longer produced, they cannot be written to any file. The remaining ERROR and FATAL messages will then be written to 
be.ERROR
, 
be.WARN
, and 
<http://be.INFO|be.INFO>
per the glog rules. References • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md
f
@Rocky it says this" BE/CN Logging in detail 
{be or cn}.INFO.log
 It primarily records various runtime behavior logs generated by BE/CN nodes, and these logs are at the 
INFO
level. For example:
@Rocky so if I set LOG_LEVEL to warn and only look at be.INFO, what logs will be emitted?
r
That specific sentence from the BE/CN Logging in detail section can be a bit misleading because it focuses on what is unique to that file (the INFO level logs). However, the "and above" behavior is confirmed in the same document by looking at the descriptions for the other BE log files and the FE section: 1. Direct "And Above" Mentions in BE Docs In the same documentation page, the sections immediately following 
be.INFO.log
explicitly state: * 
be.WARN.log
: "...stores log entries at WARNING level and above." * 
be.ERROR.log
: "...stores log entries at ERROR level and above." 2. FE Documentation Comparison The documentation for the Frontend (FE) is even more explicit about this hierarchy: * 
fe.log
: The docs state, "Compared to 
fe.log
, which records logs of all levels, 
fe.warn.log
focuses on warnings..." 3. Underlying Technical Reason (Google glog) The reason 
be.INFO.log
behaves this way (recording INFO, WARN, ERROR, and FATAL) is that the StarRocks Backend uses Google glog. The standard behavior of glog is designed so that a higher-severity message is automatically mirrored in every file of lower severity. This ensures that the 
INFO
log serves as a complete chronological record of everything the system decided to log. How to verify If you look at a live 
be.INFO.log
file, you will see lines prefixed with 
W
(Warning) or 
E
(Error). If it only recorded INFO level logs, you would only see lines starting with 
I
. Summary: While the doc says 
be.INFO.log
"primarily records... logs at the INFO level," this is meant to distinguish its purpose from the specialized WARN/ERROR files. In practice, it acts as the "master log" containing all levels permitted by your `sys_log_level`log_level` configuration`. References • administration/management/logs.md • administration/management/BE_configuration.md References • administration/management/BE_configuration.md • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/FE_configuration.md
If you set 
sys_log_level
to 
WARN
, the following logs will be emitted into the 
<http://be.INFO|be.INFO>
file: * WARNING * ERROR * FATAL The INFO level logs will be completely discarded by the system and will not appear in any file. Why this happens: 1. Level Filtering: The 
sys_log_level
configuration acts as a "gatekeeper." When set to 
WARN
, the system stops generating any log entries with a severity lower than 
WARN
. 2. Glog Hierarchy: StarRocks' Backend uses Google's 
glog
library. In 
glog
, any log message is written to its own severity file and all files of lower severity. * 
<http://be.INFO|be.INFO>
is the lowest severity file. Therefore, every single log entry that the system is currently allowed to produce (WARN, ERROR, and FATAL) is mirrored into 
<http://be.INFO|be.INFO>
. What you will see in `be.INFO`: In this scenario, 
<http://be.INFO|be.INFO>
effectively becomes a copy of 
be.WARN
. You will see lines starting with: * 
W
(for Warning) * 
E
(for Error) * 
F
(for Fatal) You will see zero lines starting with 
I
(for Info), because the level filter 
sys_log_level = WARN
prevented them from being created in the first place. Summary of file contents at `sys_log_level = WARN`: Log File | Content 
<http://be.INFO|be.INFO>
 | WARNING, ERROR, FATAL 
be.WARN
 | WARNING, ERROR, FATAL 
be.ERROR
 | ERROR, FATAL Note: Even though the documentation says 
<http://be.INFO|be.INFO>
"primarily records... logs at the INFO level," this is a description of its purpose in a default state. In practice, it always acts as the "master log" for all active levels. References • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md • administration/management/FE_configuration.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/logs.md
f
@Rocky can you please keep answers short and concise
r
Understood. If 
sys_log_level
is set to 
WARN
, the 
<http://be.INFO|be.INFO>
file will contain only WARNING, ERROR, and FATAL logs. INFO-level logs are filtered out at the source and will not appear in any file. Summary: * Emitted: WARNING, ERROR, FATAL * Not Emitted: INFO
f
@Rocky is there a discrepacy in the documentation? they mention be.WARN.log but I have be.WARNING
r
Yes, there is a minor naming discrepancy in some parts of the documentation. While some pages refer to 
be.WARN.log
, the actual files generated by the Backend (BE) process follow the Google glog naming convention. In your 
${STARROCKS_HOME}/log
directory, you will typically find: * 
<http://be.INFO|be.INFO>
 (Master log containing INFO and above) * 
be.WARNING
 (Contains WARNING and above) * 
be.ERROR
 (Contains ERROR and above) Why the confusion? 1. Framework Difference: The Frontend (FE) uses a Java logging framework (Log4j), which produces files like 
fe.log
and 
fe.warn.log
. The documentation sometimes uses these names for the BE to maintain a consistent style, even though the BE uses C++ glog. 2. Symlinks vs. Full Names: glog creates long filenames with timestamps (e.g., 
be.INFO.20230526-120000.1234
). It then creates symlinks named simply 
<http://be.INFO|be.INFO>
or 
be.WARNING
that always point to the most recent log file. Bottom line: You should look at 
be.WARNING
. It is the same thing the documentation is referring to when it says 
be.WARN.log
. References • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md • administration/management/FE_configuration.md References • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md • administration/management/FE_configuration.md References • administration/management/logs.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md
f
@Rocky my BE nodes were down and no logs show in be.WARNING of be.ERROR is this supposed? is there a way to reduce the number of logs in the be.INFO - it seems to much debug
r
1. Why 
be.WARNING
and 
be.ERROR
are empty It is common for these to be empty even during a crash. When a BE node goes down abruptly (e.g., a crash, panic, or OOM kill), the failure details are usually written to 
be.out
 instead of the standard log files. * Check 
be.out
: This file captures 
stdout
and 
stderr
. If the process crashed due to a segmentation fault or a programmatic assertion, the stack trace will be here. * Check 
dmesg
/ System Logs: If 
be.out
is also empty, the process was likely killed by the OS (Out Of Memory Killer). Run 
dmesg -T | grep -i oom
to verify. 2. How to reduce logs in 
<http://be.INFO|be.INFO>
 If 
<http://be.INFO|be.INFO>
feels like it has too much "debug" information, check these two configurations in `be.conf`: 1. Check Verbose Logging: Ensure 
sys_log_verbose_modules
is empty. If it is set to 
*
or specific modules, the BE will flood 
<http://be.INFO|be.INFO>
with low-level 
VLOG
(verbose) entries. _ _Fix*: Set 
sys_log_verbose_modules =
(empty) and restart. 2. Raise the Log Level: As you previously considered, change the global level from 
INFO
to 
WARN
. _ _Fix*: Set 
sys_log_level = WARN
. This stops the system from emitting any 
INFO
logs entirely, leaving only Warnings and Errors in your files. Summary of Recommendations: * To find out why it went down: Look at 
be.out
. * To stop the noise: Set 
sys_log_level = WARN
and ensure 
sys_log_verbose_modules
is empty. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/logs.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • faq/Sql_faq.md • administration/management/logs.md
f
@Rocky so be.INFO contains info,warn, errors, etc - that is not very clear in the docs
r
Yes, your observation is correct, and the documentation is indeed a bit ambiguous on this point. The Log Hierarchy StarRocks' Backend (BE) uses Google glog, which follows a cumulative logging pattern. Each log file contains entries for its specific level and all levels above it: * 
<http://be.INFO|be.INFO>
: Contains INFO + WARNING + ERROR + FATAL. (This is why it feels "too much"). * 
be.WARNING
: Contains WARNING + ERROR + FATAL. * 
be.ERROR
: Contains ERROR + FATAL. * 
be.FATAL
: Contains FATAL only. The documentation explicitly mentions "and above" for the Warning and Error files but omits that detail for the Info file, leading to the confusion. How to Reduce 
<http://be.INFO|be.INFO>
Volume To make 
<http://be.INFO|be.INFO>
less noisy without turning off logs entirely: 1. Check Verbose Logging: Ensure 
sys_log_verbose_modules
is empty in 
be.conf
. If it's set to 
*
, it prints massive amounts of debug-level internal info. 2. Set 
sys_log_level = WARN
: This is the most effective way to quiet the system. It prevents the BE from emitting 
INFO
logs at all. After this change: * 
<http://be.INFO|be.INFO>
will only contain WARNING and above. * The file size will drop significantly. Note on "Empty" Logs during Crashes If your BE nodes go down and both 
be.WARNING
and 
be.ERROR
are empty, it usually means the process died so suddenly it couldn't flush its logs. * Check 
be.out
: This is where the low-level crash dump or "Segmentation Fault" message usually goes. * Check 
dmesg
: If 
be.out
is also empty, the OS likely killed the process due to an OOM (Out of Memory) event. References • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md • faq/operation_maintenance_faq.md References • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md
f
@Rocky
Copy code
root@amplmkt-gcp-prod-usce1-starrocks-cluster00-be02:/opt/starrocks/logs$ grep "GlobalUpdate is too busy" be.WARNING
W20260526 13:39:29.909212 140063176296000 global.cpp:251] GlobalUpdate is too busy!
W20260526 13:47:16.210672 140063176296000 global.cpp:251] GlobalUpdate is too busy!
W20260526 13:49:54.766236 140063176296000 global.cpp:251] GlobalUpdate is too busy!
W20260526 13:52:53.562973 140063176296000 global.cpp:251] GlobalUpdate is too busy!
W20260526 13:58:02.352892 140063176296000 global.cpp:251] GlobalUpdate is too busy!
root@amplmkt-gcp-prod-usce1-starrocks-cluster00-be02:/opt/starrocks/logs$ grep "GlobalUpdate is too busy" <http://be.INFO|be.INFO>
if your theory was correct this should have showed the same, but it didnt
r
You are absolutely right to call that out. Based on your grep results, it appears the behavior is different from what one would expect from a standard cumulative log. The most likely explanation for why you see the logs in 
be.WARNING
but not in 
<http://be.INFO|be.INFO>
is Log Rotation Mismatch: 1. Symlink vs. Reality: 
<http://be.INFO|be.INFO>
and 
be.WARNING
are typically symlinks to the latest log file (e.g., 
be.INFO.20260526-130000.1234
). 2. Rotation Frequency: Because your 
<http://be.INFO|be.INFO>
is "too much debug" (very noisy), it reaches the rotation size limit (default 1GB) much faster than 
be.WARNING
. 3. The Gap: If 
<http://be.INFO|be.INFO>
rotated at 13:55, the current 
<http://be.INFO|be.INFO>
symlink points to a file that only contains logs after 13:55. Meanwhile, since there are fewer warnings, the 
be.WARNING
symlink might still point to a file that started hours ago and contains the 13:39 events. To verify this, try running:
Copy code
bash
ls -l <http://be.INFO|be.INFO> be.WARNING
You will likely see they point to different timestamps. To find the missing logs, you would need to grep the older rotated INFO files: 
grep "GlobalUpdate is too busy" be.INFO.log.20260526*
. Regarding the "GlobalUpdate is too busy" error: This specific warning (
global.cpp:251
) indicates that the Primary Key update/delete mechanism is under heavy pressure. * It usually happens when you are loading data into Primary Key tables faster than the BE can apply the "Global Index" updates. * If this is happening alongside BE nodes going down, it’s a strong signal that the nodes are being overwhelmed by write load or memory pressure from PK index maintenance. Recommendation: Since you want to reduce the noise in `be.INFO`: 1. Check 
sys_log_level
: If you change this to 
WARN
in 
be.conf
, the BE will stop writing INFO-level logs entirely. 2. Check 
sys_log_verbose_modules
: Ensure this is empty. If it's set to 
*
, it prints "VLOG" entries which are even noisier than standard INFO logs. References • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md References • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md
f
@Rocky what files are really important to ingest logs for both fe and be? only the critically important ones
r
For a production environment where you want to minimize noise while keeping the "critically important" information for troubleshooting and monitoring, you should focus on these files: 1. Backend (BE) Logs * 
be.WARNING
: Most Critical. This captures all 
WARNING
, 
ERROR
, and 
FATAL
logs. In StarRocks, many critical system-pressure signals (like your 
GlobalUpdate is too busy
) are logged as 
WARNING
, not 
ERROR
. * 
be.out
: Essential for Crashes. When a BE process crashes or encounters a segmentation fault, the stack trace is written here, not to the standard 
.WARNING
or 
.INFO
files. _ _Note on `be.INFO`:* Usually too high-volume for ingestion unless you are actively debugging. 2. Frontend (FE) Logs * 
fe.warn.log
: Most Critical. Contains all 
WARN
and 
ERROR
level events for the FE (e.g., metadata issues, connection timeouts, leader election problems). * 
fe.audit.log
: Performance & Security. This records every query, who ran it, how long it took, and how many rows were returned. _ _Tip:* Set 
audit_log_json_format = true
in 
fe.conf
to make this much easier to ingest into ELK/Loki. * 
fe.out
: Essential for JVM Crashes. Captures standard output/error for the Java process, including JVM OutOfMemory errors and startup failures. Summary Checklist for Ingestion File | Importance | Why? 
be.WARNING
 | Critical | Captures errors and system pressure signals. 
be.out
 | Critical | The only place to find crash/segfault details. 
fe.warn.log
 | Critical | Main FE health indicator. 
fe.audit.log
 | High | Necessary for slow query analysis and auditing. 
fe.out
 | High | Captures JVM-level crashes and startup errors. Pro-tip: If you find 
be.WARNING
is still too noisy because of specific recurring messages, you can't easily filter them at the source, but you should definitely avoid ingesting 
<http://be.INFO|be.INFO>
as it will likely be 10x-100x larger than the 
WARNING
file. References • administration/management/logs.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • sql-reference/information_schema/be_logs.md References • administration/management/logs.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • sql-reference/information_schema/be_logs.md • administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • administration/management/resource_management/Resource_Group.md • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md • administration/management/monitoring/Monitor_and_Alert.md
f
@Rocky if I set LOG_LEVEL to WARN what will fe.log have?
r
If you set 
sys_log_level = WARN
in your 
fe.conf
, the behavior of 
fe.log
changes significantly: What 
fe.log
will contain: * Only 
WARN
, 
ERROR
, and 
FATAL
messages. * Because the system-wide logging threshold is now 
WARN
, the standard 
INFO
level messages (which make up 90% of a healthy 
fe.log
) will be discarded before they are written. * Effectively: 
fe.log
and 
fe.warn.log
will become nearly identical, as both will now only capture warning-level events and above. What you will LOSE in `fe.log`: If you set the level to 
WARN
, you will no longer see the following "healthy" signals in `fe.log`: * Query Lifecycle INFO: "Execute query," "Finished query," and "Store profile" messages. * Heartbeat INFO: Logs showing successful heartbeats between FE and BEs. * Metadata INFO: Details about image creation (checkpoints), BDBJE replication status, and schema change progress. * Connection INFO: Messages about new client connections or disconnections. What is NOT affected: 1. 
fe.audit.log
: This file is not controlled by 
sys_log_level
. It will continue to record every SQL query, its execution time, and its status (even if successful) regardless of your system log level. 2. 
fe.out
: This captures the direct standard output of the Java process (like JVM startup parameters and fatal crash stack traces) and is unaffected by log level settings. Recommendation If your goal is to reduce disk usage without losing the ability to see what the system is doing, it is usually better to leave 
sys_log_level
at 
INFO
but: * Reduce 
sys_log_roll_num
: Keep only 3–5 rotated files instead of 10. * Enable 
sys_log_enable_compress = true
: This will Gzip the rotated 
fe.log
files, reducing their size by ~90% on disk. References • administration/management/logs.md • administration/management/FE_configuration.md References • administration/management/logs.md • administration/management/FE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • administration/management/resource_management/Resource_Group.md • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/FE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • administration/management/resource_management/Resource_Group.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/management/logs.md
2 Views
Open in Slack
PreviousNext
Powered by