http://starrocks.io logo
Join Slack
Powered by
Hey <@U0A71G31CDV>, What's the best way to parse ...
# questions-and-troubleshooting
f
Hey @Rocky, What's the best way to parse NON-JSON BE node logs with something like Fluent Bit to then ingest to Grafana Loki? Is there a pattern we can follow or what's the recommended approach?
r
StarRocks BE node logs use the glog (Google Logging) format, which is a specialized plaintext format. To ingest these into Grafana Loki via Fluent Bit, the recommended approach is to use a Regex Parser combined with a Multiline Filter (to handle stack traces) and the Loki Output Plugin. 1. Identify the Log Pattern A typical StarRocks BE log line (e.g., 
<http://be.INFO|be.INFO>
) looks like this:
Copy code
text
I0520 17:02:05.123456 12345 tablet_manager.cpp:123] Successfully loaded tablet 10001
* 
I
: Log Level (
I
=INFO, 
W
=WARN, 
E
=ERROR, 
F
=FATAL) * 
0520
: Date (MMDD) * 
17:02:05.123456
: Timestamp (HHMMSS.usec) * 
12345
: Thread ID (TID) * 
tablet_manager.cpp:123
: Source file and line number * 
]
: Separator * 
Successfully...
: The log message 2. Recommended Fluent Bit Configuration A. Parser Definition (
parsers.conf
) You need a regex that handles the combined Level/Date field and extracts the metadata.
Copy code
ini
[PARSER]
    Name        starrocks_be
    Format      regex
    # Regex captures Level, Date+Time, TID, File, and Message
    Regex       ^(?<level>[IWEF])(?<time>\d{4}\s+\d{2}:\d{2}:\d{2}\.\d{6})\s+(?<tid>\d+)\s+(?<file>[^:]+:(?:\d+))\]\s+(?<log>.*)$
    Time_Key    time
    Time_Format %m%d %H:%M:%S.%L
    # Note: glog doesn't include the year by default. 
    # Fluent Bit will assume the current year.
B. Multiline Configuration Since BE logs often contain C++ stack traces or multiline error messages, use a Multiline Parser to avoid breaking them into separate records.
Copy code
ini
[MULTILINE_PARSER]
    name          multiline_starrocks_be
    type          regex
    flush_timeout 1000
    # A new log entry starts with Level (I/W/E/F) followed by MMDD
    rule          "start_state"   "/^[IWEF]\d{4}\s+\d{2}:\d{2}:\d{2}\./"  "cont"
    rule          "cont"          "/^(?![IWEF]\d{4}\s+\d{2}:\d{2}:\d{2}\.)/" "cont"
C. Input and Output (
fluent-bit.conf
)
Copy code
ini
[INPUT]
    Name              tail
    Path              /path/to/starrocks/be/log/be.INFO
    Tag               <http://starrocks.be|starrocks.be>
    multiline.parser  multiline_starrocks_be

[FILTER]
    Name         parser
    Match        <http://starrocks.be|starrocks.be>
    Key_Name     log
    Parser       starrocks_be
    Reserve_Data True

[OUTPUT]
    Name         loki
    Match        <http://starrocks.be|starrocks.be>
    Host         loki-host
    Port         3100
    Labels       job=starrocks, component=be, level=$level
    Auto_Labels  on
3. Best Practices 1. Enable Timezones: In 
be.conf
, set 
sys_log_timezone = true
. This adds the timezone offset to the log prefix, which helps Fluent Bit and Loki align timestamps correctly if your servers aren't in UTC. 2. Log Level Labels: Map the extracted 
level
field to a Loki label. This allows you to quickly filter by 
level="E"
in Grafana. 3. Handle Year Wraparound: Because glog only provides 
MMDD
, ensure your Fluent Bit server's system clock is accurate. If you are re-processing old logs from a previous year, Fluent Bit might incorrectly tag them with the current year. 4. Audit Logs: If you also need to parse FE Audit logs (
fe.audit.log
), those are space-separated or CSV-like and require a separate parser. However, for BE, the glog regex above is the standard pattern. References • sql-reference/information_schema/be_logs.md • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/BE_configuration.md
f
@Rocky is there a recommended approach besides using Fluent Bit?
r
Besides Fluent Bit, the two most common and recommended approaches for ingesting StarRocks BE logs into Loki are Vector and Promtail (or the newer Grafana Alloy). Since BE logs use the glog format (which is not natively JSON), both tools require a regex-based parsing step. 1. Vector (Recommended for Performance) Vector is often preferred because it is built in Rust (high performance) and uses VRL (Vector Remap Language), which is very powerful for handling the quirky glog timestamp format (which lacks a year). Example 
vector.toml
configuration:
Copy code
toml
[sources.be_logs]
type = "file"
include = ["/opt/starrocks/be/log/be.INFO*"]
multiline.start_pattern = '^[IWEF]\d{4}'
multiline.condition_pattern = '^[IWEF]\d{4}'
multiline.mode = "continue_through"

[transforms.parse_glog]
type = "remap"
inputs = ["be_logs"]
source = '''
# 1. Parse the glog structure
. |= parse_regex!(.message, r'^(?P<level>[IWEF])(?P<timestamp>\d{4}\s+\d{2}:\d{2}:\d{2}\.\d{6})\s+(?P<tid>\d+)\s+(?P<file>[^:]+:(?:\d+))\]\s+(?P<msg>.*)$')

# 2. Fix the missing year in glog
# Glog uses MMDD. We prepend the current year to make it ISO-compliant.
current_year = to_string(now())
current_year = slice!(current_year, 0, 4)
.timestamp = current_year + .timestamp

# 3. Map levels to words for easier Loki filtering
.level = replace(.level, "I", "info")
.level = replace(.level, "W", "warn")
.level = replace(.level, "E", "error")
.level = replace(.level, "F", "fatal")
'''

[sinks.loki]
type = "loki"
inputs = ["parse_glog"]
endpoint = "<http://loki:3100>"
encoding.codec = "json"
labels.job = "starrocks-be"
labels.level = "{{level}}"
────────── 2. Promtail (The "Native" Loki Agent) If you prefer to stay within the official Grafana ecosystem, Promtail is the standard choice. It uses a pipeline-based configuration. Example `promtail-config.yaml`:
Copy code
yaml
scrape_configs:
- job_name: starrocks-be
  static_configs:
  - targets: [localhost]
    labels:
      job: starrocks-be
      __path__: /opt/starrocks/be/log/be.INFO*
  pipeline_stages:
  # 1. Multiline stage to catch stack traces
  - multiline:
      firstline: '^[IWEF]\d{4}'
      max_wait_time: 3s
  # 2. Regex stage to extract metadata
  - regex:
      expression: '^(?P<level>[IWEF])(?P<timestamp>\d{4}\s+\d{2}:\d{2}:\d{2}\.\d{6})\s+(?P<tid>\d+)\s+(?P<file>[^:]+:(?:\d+))\]\s+(?P<msg>.*)$'
  # 3. Add labels for Loki
  - labels:
      level:
      file:
  # 4. Optional: Use the extracted timestamp
  - timestamp:
      source: timestamp
      format: "MMDD HH:mm:ss.SSSSSS"
────────── Key Comparison & Tips Feature | Vector | Promtail Performance | Extremely high (Rust) | Moderate (Go) Parsing Logic | VRL (Rich scripting) | Pipeline Stages (YAML) Timestamp | Easy to append "Year" via VRL | Uses system year by default Footprint | Small | Small to Moderate Important Configuration Tip: In your StarRocks 
be.conf
, make sure 
sys_log_timezone = true
is set. This ensures the logs include timezone information, which prevents timestamp drift when your log collector (Vector/Promtail) and the StarRocks process are in different environments or if the system clock isn't set to UTC. Wait, what about JSON? While you asked for non-JSON, it's worth noting that FE logs can be switched to native JSON via 
sys_log_format = json
(as of v3.2.10). However, BE logs remain in the glog format for the foreseeable future, so the Regex/Vector patterns above are the correct "best practice" for BE. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/logs.md
f
@Rocky are the log files called be.WARNING or be.WARN.log? I think the docs are wrong
r
You are correct to be suspicious—there is a slight inconsistency in the documentation regarding Backend (BE) log filenames. The Short Answer: On a standard StarRocks BE node, the log files are typically called: * 
<http://be.INFO|be.INFO>
 * 
be.WARNING
 * 
be.ERROR
 These are actually symbolic links created by the glog (Google Logging) library that StarRocks uses for its C++ backend. They point to the latest timestamped log file (e.g., 
be.hostname.user.log.INFO.20240520-170205.12345
). Why the docs might look "wrong" In some recent versions of the documentation (specifically the 3.5.x "Logs" page), the files are referred to as 
be.INFO.log
or 
be.WARN.log
. This appears to be a documentation error or an attempt to align the naming convention with the Frontend (FE) logs. * FE (Java) uses log4j, which produces: 
fe.log
and 
fe.warn.log
. * BE (C++) uses glog, which produces: 
<http://be.INFO|be.INFO>
and 
be.WARNING
. Which one should you use for Fluent Bit/Loki? When configuring Fluent Bit, Vector, or Promtail, you should look for the following patterns in your 
sys_log_dir
(default is 
BE_HOME/log
): 1. 
<http://be.INFO|be.INFO>
: This is the primary log file containing all levels (INFO and up). Most users ingest this one. 2. 
be.WARNING
: Contains only WARNING and ERROR levels. 3. 
be.ERROR
: Contains only ERROR levels. Recommended approach for your Fluent Bit config: Point your 
tail
input to the symbolic link or use a wildcard for the actual log files:
Copy code
ini
[INPUT]
    Name        tail
    # Using the symlink (standard way)
    Path        /path/to/starrocks/be/log/be.INFO
    # OR using a wildcard to catch rotated files
    # Path      /path/to/starrocks/be/log/be.*.log.INFO.*
Summary of Differences Component | Library | Primary Log | Warning Log FE (Frontend) | Log4j | 
fe.log
| 
fe.warn.log
BE (Backend) | Glog | 
<http://be.INFO|be.INFO>
 | 
be.WARNING
 Note: If you are using the StarRocks Kubernetes Operator, it occasionally maps these differently in volumes, but the internal process still generates 
<http://be.INFO|be.INFO>
and 
be.WARNING
. References • administration/management/logs.md • administration/management/BE_configuration.md
Open in Slack
PreviousNext
Powered by