Hi everyone I observed a very strange behaviour on...
# fluent-bit
h
Hi everyone I observed a very strange behaviour on a lot of (more than 20) pods of FluentBit (of 46 pods of the DS).
Indeed I can see the same pattern of logs
Copy code
[2025/06/25 20:13:32] [engine] caught signal (SIGHUP)
[2025/06/25 20:13:32] [ info] reloading instance pid=1 tid=0x7f37941dae40
[2025/06/25 20:13:32] [ info] [reload] stop everything of the old context
[2025/06/25 20:13:32] [ warn] [engine] service will shutdown when all remaining tasks are flushed
[2025/06/25 20:13:32] [ info] [reload] start everything
then nothing else
tail inputs have collected no data things .. likely they have been hang since
once I killed the pod, the new one is created and logs are collected again
the FluentBit processes seem to do anything and stuck for hours
p
Looks like it's stuck in a reload
Did you trigger one?
I recall seeing a recent issue around this so maybe check GitHub
h
it's sleeping
Copy code
# cat /proc/1548315/stack 
[<0>] hrtimer_nanosleep+0x95/0x120
[<0>] common_nsleep+0x40/0x50
[<0>] __x64_sys_clock_nanosleep+0xc7/0x130
[<0>] do_syscall_64+0x35/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
p
Sure, but did you trigger a reload as that seems to be what it's doing? I'm sure there was a recent issue or something about this on reload
h
yes ... we trigger the hot reload when the certificate used by FluentBit has been changed
for the kafka input, we are using mtls and the certificate is renewed every (month .. If I recall well) .. so the hot reload is triggered when the secret containing the certificate is updated
we had 24 pods stuck is this state (of 46 pods of the DS) with the same pattern logs 👆
Copy code
[ info] [reload] start everything
p
https://github.com/fluent/fluent-bit/issues/9354 maybe, I think @Phillip Whelan was looking into it
h
yes I saw ..it's similar but we don't have the logs of pause and it stucks after "reload start everything"
p
For some reason I thought there was something more recent but cannot find it
h
this is the first time we got this issue ..
even the hot reload have been triggered before
nothing changed on the system/kube side afaik
p
yeah but race conditions are inherently like that
😰 1
😃 1