Hendrik
06/12/2025, 11:31 AMThread 7 (Thread 0x73c6f880 (LWP 2934) "flb-out-prometh"):
#0 0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1 0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2 0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x7325ba00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3 mk_event_wait (loop=0x7325ba00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4 0x004ce0cc in output_thread (data=0x768c6e00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_output_thread.c:257
#5 0x004dd8cc in step_callback (data=0x768d55e0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6 0x7693cdfc in start_thread () from /lib/libc.so.6
#7 0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 6 (Thread 0x74482880 (LWP 2933) "flb-out-prometh"):
#0 0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1 0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2 0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x7325ba50) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3 mk_event_wait (loop=0x7325ba50) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4 0x004ce0cc in output_thread (data=0x768c6c40) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_output_thread.c:257
#5 0x004dd8cc in step_callback (data=0x768d5500) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6 0x7693cdfc in start_thread () from /lib/libc.so.6
#7 0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 5 (Thread 0x74d4f880 (LWP 2932) "flb-in-process_"):
#0 0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1 0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2 0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x74d80520) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3 mk_event_wait (loop=0x74d80520) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4 0x004ca69a in input_thread (data=0x768be3c0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_input_thread.c:407
#5 0x004dd8cc in step_callback (data=0x768d5490) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6 0x7693cdfc in start_thread () from /lib/libc.so.6
#7 0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 4 (Thread 0x755b9880 (LWP 2931) "flb-in-node_exp"):
#0 0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1 0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2 0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x76f145c0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3 mk_event_wait (loop=0x76f145c0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4 0x004ca69a in input_thread (data=0x768be280) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_input_thread.c:407
#5 0x004dd8cc in step_callback (data=0x768d53b0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6 0x7693cdfc in start_thread () from /lib/libc.so.6
#7 0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 3 (Thread 0x75dd5880 (LWP 2930) "flb-logger"):
#0 0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1 0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2 0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x76f140d0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3 mk_event_wait (loop=0x76f140d0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4 0x004c0568 in log_worker_collector (data=0x768db460) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_log.c:163
#5 0x004dd8cc in step_callback (data=0x768d4e00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6 0x7693cdfc in start_thread () from /lib/libc.so.6
#7 0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 2 (Thread 0x765fe880 (LWP 2929) "flb-pipeline"):
#0 0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--c
#1 0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2 0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x76f140a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3 mk_event_wait (loop=loop@entry=0x76f140a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4 0x004d6ac2 in flb_engine_start (config=config@entry=0x76c24000) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_engine.c:971
#5 0x004bf9e8 in flb_lib_worker (data=0x74db33a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_lib.c:835
#6 0x7693cdfc in start_thread () from /lib/libc.so.6
#7 0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 1 (Thread 0x76f40020 (LWP 19666) "fluent-bit"):
#0 0x76901ae4 in __libc_do_syscall () from /lib/libc.so.6
#1 0x7693a5f6 in __futex_abstimed_wait_common () from /lib/libc.so.6
#2 0x7693a6b4 in __futex_abstimed_wait_cancelable64 () from /lib/libc.so.6
#3 0x7693e2a6 in __pthread_clockjoin_ex () from /lib/libc.so.6
#4 0x7693e1d8 in pthread_join () from /lib/libc.so.6
#5 0x004bf7fe in flb_stop (ctx=0x74db33a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_lib.c:978
#6 0x00463d32 in flb_main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/fluent-bit.c:1497
#7 0x769017c6 in __libc_start_call_main () from /lib/libc.so.6
#8 0x76901870 in __libc_start_main () from /lib/libc.so.6
#9 0x004623d4 in _start ()
Let me know if you need more information or if you have hints on how to resolve this problem 🙂 Thanks in advance!Pat
06/12/2025, 11:32 AMHendrik
06/12/2025, 11:56 AMenv:
serial_number: XXXXXXXXXX
syslog_log_file: "/media/card/syslog/DevLogAll.log"
fluent_bit_log_file: "/var/volatile/log/fluent-bit.log"
prometheus_hostname: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
prometheus_port: 443
loki_hostname: XXXXXXXXXXXXXXXXXXXXXXXXXXXX
loki_port: 443
includes:
- interval-conf.yaml
- fluent-bit-metrics.yaml
- system-metrics.yaml
- process-metrics.yaml
- firmware-logs.yaml
service:
flush: 10s
log_level: warn
log_file: ${fluent_bit_log_file}
parsers_file: /etc/fluent-bit/XXXXXXXXXX.conf
pipeline:
inputs:
- name: opentelemetry
listen: 127.0.0.1
port: 4317
successful_response_code: 200
tag: otel_metrics
storage.type: memory
mem_buf_limit: 1MB
outputs:
- name: prometheus_remote_write
match: "*_metrics"
host: ${prometheus_hostname}
port: ${prometheus_port}
tls.ca_file: "/media/card/certificates/prometheus/ca.crt"
tls.crt_file: "/media/card/certificates/prometheus/client.crt"
tls.key_file: "/media/card/certificates/prometheus/client.key"
uri: "/api/v1/write"
log_response_payload: True
tls: On
tls.verify: On
add_label:
- serial_number ${serial_number}
- namespace controller-health-monitoring
retry_limit: 3
- name: loki
match: "*_logs"
host: ${loki_hostname}
port: ${loki_port}
uri: "/loki/api/v1/push"
tls.ca_file: "/media/card/certificates/loki/ca.crt"
tls.crt_file: "/media/card/certificates/loki/client.crt"
tls.key_file: "/media/card/certificates/loki/client.key"
tls.verify: On
tls: On
# These keys are set as labels.
label_keys: $component, $source_file, $source_file_line, $commit_hash, $log_level, $serial_number, $namespace, $service
# All keys that are not removed here will be concatenated to form the log message. They are set as labels instead through the line above.
remove_keys: $component, $source_file, $source_file_line, $commit_hash, $log_level, $serial_number, $namespace, $service
line_format: key_value
retry_limit: 3
• Let me know if you need any of the included configuration files.
• We're running fluent-bit 4.0.0 on armv7l on Yocto Poky Kirkstone.
• Here is the command line: /usr/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.yaml --enable-hot-reload --daemon
• No, we're compiling the binaries via Yocto.
• We upgraded multiple times in the past time. Latest version is 4.0.3. If we would upgrade now from 4.0.0, we would need to wait for a month to get into a hanging state again. Should we do that? Is it necessary or is 4.0.0 new enough?
• This hanging only happens after some months of runtime.
• Due to https://github.com/fluent/fluent-bit/issues/9547 we're sending SIGHUP daily to the fluent-bit process to reload and prevent high CPU usage.Pat
06/12/2025, 12:15 PMPat
06/12/2025, 12:16 PMPat
06/12/2025, 12:18 PMHendrik
06/12/2025, 12:23 PMPat
06/12/2025, 12:24 PMHendrik
06/12/2025, 1:25 PMPat
06/12/2025, 1:44 PM