Hello, We have a problem with fluent-bit 4.1.x running on RHEL10. The agent crash with a core-dump. • It looks like the problem is in the systemd plugin. • If we delete all the content under

storage.path

and restart the agent, it works and process data for a while and then crash with a core-dump. • If we try to start the agent again after this without deleting the chunk/db information under

storage.path

it crash right away with a core dump (fluent-bit gets terminated with signal SIGABRT, Aborted) • I suspect that something corrupts the chunk data and after that the agent is not able to start again. • What do you need to debug this bug? • I have core dumps generated with 4.1.0 and 4.1.1 and data under

storage.path

that can be used to reproduce this problem. This is the backtrace from 4.1.1:

(gdb) backtrace 
#0  0x00007fd5df4bd9dc in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007fd5df467a96 in raise () from /lib64/libc.so.6
#2  0x00007fd5df44f8fa in abort () from /lib64/libc.so.6
#3  0x00000000004ae47d in flb_signal_handler (signal=11) at /root/rhel10-test/fluent-bit-4.1.1/src/fluent-bit.c:636
#4  <signal handler called>
#5  0x00000000011d0e84 in ZSTD_freeDDict (ddict=0x1) at /root/rhel10-test/fluent-bit-4.1.1/lib/zstd-1.5.7/lib/decompress/zstd_ddict.c:215
#6  0x00007fd5df9ad609 in ZSTD_freeDCtx () from /lib64/libzstd.so.1
#7  0x00007fd5e01e5b09 in journal_file_data_payload.isra () from /lib64/libsystemd.so.0
#8  0x00007fd5e0142d33 in sd_journal_enumerate_data () from /lib64/libsystemd.so.0
#9  0x00000000007466ea in in_systemd_collect (ins=0x390d8d60, config=0x390a7490, in_context=0x7fd5d0001090) at /root/rhel10-test/fluent-bit-4.1.1/plugins/in_systemd/systemd.c:387
#10 0x0000000000746b07 in in_systemd_collect_archive (ins=0x390d8d60, config=0x390a7490, in_context=0x7fd5d0001090) at /root/rhel10-test/fluent-bit-4.1.1/plugins/in_systemd/systemd.c:512
#11 0x0000000000504fcd in input_collector_fd (fd=39, ins=0x390d8d60) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_input_thread.c:166
#12 0x0000000000505b0a in engine_handle_event (fd=39, mask=1, ins=0x390d8d60, config=0x390a7490) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_input_thread.c:181
#13 input_thread (data=0x7fd5d801f4e0) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_input_thread.c:420
#14 0x000000000057ca77 in step_callback (data=0x7fd5d8024b70) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_worker.c:43
#15 0x00007fd5df4bbb68 in start_thread () from /lib64/libc.so.6
#16 0x00007fd5df52c6bc in clone3 () from /lib64/libc.so.6

What target are you installing? There is no RHEL 10 RPM but there are versions built for AlmaLinux and RockyLinux so which one is it? Please do not be installing a CentOS 10 RPM as that will break now that Red Hat have made CentOS Stream upstream of RHEL so dependencies will often break.

You can use https://github.com/chronosphereio/calyptia-chunk-inspector to dump the content

Thanks, I will do all this ..... the package we use is from the almalinux repo at packages.fluentbit.io. But I have compiled 4.1.0 and 4.1.1 from source to activate

FLB_DEBUG

and other development options so I could get more information from the coredump files and they have the same problem as the rpm package.

It looks like the pointer here is invalid (not null): https://github.com/fluent/fluent-bit/blob/v4.1.1/lib/zstd-1.5.7/lib/decompress/zstd_ddict.c#L215 Unfortunately that's a dependency of a dependency so might be worth checking there too.