Hi, I'm currently analyzing a hanging shutdown of ...
# fluent-bit
h
Hi, I'm currently analyzing a hanging shutdown of a fluent-bit process. Is there someone available for help with more knowledge in the project? It does not react to SIGTERM signals by not shutting down. Here are the backtraces of the threads:
Copy code
Thread 7 (Thread 0x73c6f880 (LWP 2934) "flb-out-prometh"):
#0  0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1  0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2  0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x7325ba00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3  mk_event_wait (loop=0x7325ba00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4  0x004ce0cc in output_thread (data=0x768c6e00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_output_thread.c:257
#5  0x004dd8cc in step_callback (data=0x768d55e0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6  0x7693cdfc in start_thread () from /lib/libc.so.6
#7  0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (Thread 0x74482880 (LWP 2933) "flb-out-prometh"):
#0  0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1  0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2  0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x7325ba50) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3  mk_event_wait (loop=0x7325ba50) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4  0x004ce0cc in output_thread (data=0x768c6c40) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_output_thread.c:257
#5  0x004dd8cc in step_callback (data=0x768d5500) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6  0x7693cdfc in start_thread () from /lib/libc.so.6
#7  0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 5 (Thread 0x74d4f880 (LWP 2932) "flb-in-process_"):
#0  0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1  0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2  0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x74d80520) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3  mk_event_wait (loop=0x74d80520) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4  0x004ca69a in input_thread (data=0x768be3c0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_input_thread.c:407
#5  0x004dd8cc in step_callback (data=0x768d5490) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6  0x7693cdfc in start_thread () from /lib/libc.so.6
#7  0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (Thread 0x755b9880 (LWP 2931) "flb-in-node_exp"):
#0  0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1  0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2  0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x76f145c0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3  mk_event_wait (loop=0x76f145c0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4  0x004ca69a in input_thread (data=0x768be280) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_input_thread.c:407
#5  0x004dd8cc in step_callback (data=0x768d53b0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6  0x7693cdfc in start_thread () from /lib/libc.so.6
#7  0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (Thread 0x75dd5880 (LWP 2930) "flb-logger"):
#0  0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
#1  0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2  0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x76f140d0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3  mk_event_wait (loop=0x76f140d0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4  0x004c0568 in log_worker_collector (data=0x768db460) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_log.c:163
#5  0x004dd8cc in step_callback (data=0x768d4e00) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_worker.c:43
#6  0x7693cdfc in start_thread () from /lib/libc.so.6
#7  0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0x765fe880 (LWP 2929) "flb-pipeline"):
#0  0x76901ae6 in __libc_do_syscall () from /lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--c
#1  0x7698dc60 in epoll_wait () from /lib/libc.so.6
#2  0x006e652a in _mk_event_wait_2 (timeout=-1, loop=0x76f140a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event_epoll.c:444
#3  mk_event_wait (loop=loop@entry=0x76f140a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/lib/monkey/mk_core/mk_event.c:207
#4  0x004d6ac2 in flb_engine_start (config=config@entry=0x76c24000) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_engine.c:971
#5  0x004bf9e8 in flb_lib_worker (data=0x74db33a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_lib.c:835
#6  0x7693cdfc in start_thread () from /lib/libc.so.6
#7  0x7698d830 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0x76f40020 (LWP 19666) "fluent-bit"):
#0  0x76901ae4 in __libc_do_syscall () from /lib/libc.so.6
#1  0x7693a5f6 in __futex_abstimed_wait_common () from /lib/libc.so.6
#2  0x7693a6b4 in __futex_abstimed_wait_cancelable64 () from /lib/libc.so.6
#3  0x7693e2a6 in __pthread_clockjoin_ex () from /lib/libc.so.6
#4  0x7693e1d8 in pthread_join () from /lib/libc.so.6
#5  0x004bf7fe in flb_stop (ctx=0x74db33a0) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/flb_lib.c:978
#6  0x00463d32 in flb_main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/fluent-bit/4.0.0-r3/git/src/fluent-bit.c:1497
#7  0x769017c6 in __libc_start_call_main () from /lib/libc.so.6
#8  0x76901870 in __libc_start_main () from /lib/libc.so.6
#9  0x004623d4 in _start ()
Let me know if you need more information or if you have hints on how to resolve this problem 🙂 Thanks in advance!
p
Config and version would help (I'm guessing 4.0.0)? What target are you using/running on? How are you triggering it? Are you using the official binaries? Does it happen with the latest version?
👀 1
h
Copy code
env:
  serial_number: XXXXXXXXXX
  syslog_log_file: "/media/card/syslog/DevLogAll.log"
  fluent_bit_log_file: "/var/volatile/log/fluent-bit.log"

  prometheus_hostname: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  prometheus_port: 443

  loki_hostname: XXXXXXXXXXXXXXXXXXXXXXXXXXXX
  loki_port: 443

includes:
  - interval-conf.yaml
  - fluent-bit-metrics.yaml
  - system-metrics.yaml
  - process-metrics.yaml
  - firmware-logs.yaml

service:
  flush: 10s
  log_level: warn
  log_file: ${fluent_bit_log_file}
  parsers_file: /etc/fluent-bit/XXXXXXXXXX.conf

pipeline:
  inputs:
    - name: opentelemetry
      listen: 127.0.0.1
      port: 4317
      successful_response_code: 200
      tag: otel_metrics
      storage.type: memory
      mem_buf_limit: 1MB

  outputs:
    - name: prometheus_remote_write
      match: "*_metrics"
      host: ${prometheus_hostname}
      port: ${prometheus_port}
      tls.ca_file: "/media/card/certificates/prometheus/ca.crt"
      tls.crt_file: "/media/card/certificates/prometheus/client.crt"
      tls.key_file: "/media/card/certificates/prometheus/client.key"
      uri: "/api/v1/write"
      log_response_payload: True
      tls:  On
      tls.verify: On
      add_label:
        - serial_number ${serial_number}
        - namespace controller-health-monitoring
      retry_limit: 3

    - name: loki
      match: "*_logs"
      host: ${loki_hostname}
      port: ${loki_port}
      uri: "/loki/api/v1/push"
      tls.ca_file: "/media/card/certificates/loki/ca.crt"
      tls.crt_file: "/media/card/certificates/loki/client.crt"
      tls.key_file: "/media/card/certificates/loki/client.key"
      tls.verify: On
      tls: On
      # These keys are set as labels.
      label_keys: $component, $source_file, $source_file_line, $commit_hash, $log_level, $serial_number, $namespace, $service
      # All keys that are not removed here will be concatenated to form the log message. They are set as labels instead through the line above.
      remove_keys: $component, $source_file, $source_file_line, $commit_hash, $log_level, $serial_number, $namespace, $service
      line_format: key_value
      retry_limit: 3
• Let me know if you need any of the included configuration files. • We're running fluent-bit 4.0.0 on armv7l on Yocto Poky Kirkstone. • Here is the command line:
/usr/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.yaml --enable-hot-reload --daemon
• No, we're compiling the binaries via Yocto. • We upgraded multiple times in the past time. Latest version is 4.0.3. If we would upgrade now from 4.0.0, we would need to wait for a month to get into a hanging state again. Should we do that? Is it necessary or is 4.0.0 new enough? • This hanging only happens after some months of runtime. • Due to https://github.com/fluent/fluent-bit/issues/9547 we're sending SIGHUP daily to the fluent-bit process to reload and prevent high CPU usage.
p
Ah, Yocto
Without knowing the cause I cannot say if 4.0.3 will resolve it - is it possible to accelerate the cause? e.g. send more data, etc.
https://github.com/fluent/fluent-bit/issues/9547 is only for the process-exporter plugin which you don't seem to be using? The related issue was closed by https://github.com/fluent/fluent-bit/pull/9549 it seems too
h
I guess, there is no way to accelerate the trigger because we also don't know the cause at the moment. Is there anything that would help analyzing with the currently attached GDB?
p
It looks like everything is waiting to be honest so maybe a livelock but hard to say. Are you sure you need to reload daily?
h
A livelock should be discoverable by high CPU usage, right? That isn't the case. It seems to me that it is some kind of deadlock. Is it somehow observable which plugin blocks?
p
Yeah I more meant it looks like stuff is waiting on notification but that might just be a snapshot of when the sigterm happens, it also looks like there is stack corruption too which can invalidate everything too.