Hey all, I wanted to clarify few things regarding ...
# fluent-bit
a
Hey all, I wanted to clarify few things regarding the
kubernetes_events
input plugin. I have noticed at some points the plugin does not collect the events as it supposed to do, and when I turn on the debug logs of the input plugin I noticed the the
[debug] [input coro] destroy coro_id=xxx
being logged continuously (Refer to the attached flb-k8s-events-input-debug.log). Has anyone else experienced the same or does anyone have any idea of why this occurs and how to solve the issue? Also I noticed in the debug logs that
[debug] [input:kubernetes_events:kubernetes_events.0] could not process payload, incomplete or bad formed JSON:
lines are getting continuously logged as well, before the burst of destry coro_id logs. I have looked at the source code and realised it is due to the chunks streamed to the input plugin might not always be in complete JSON format. What I wanted to clarify on this is, are we buffering the incomplete chunks so that those events won't be dropped from the events pipeline? If so, would you be able to explain or point me out to the code? Thanks in advance.
p
@Phillip Whelan is probably the man to ask
a
Thanks for the reply @Pat @Phillip Whelan would you be able to help?
To add more context to this issue, I was using Fluent-Bit 3.2.4, and SQLDB for kubernetes_events input plugin was not set up due to the bug (https://github.com/fluent/fluent-bit/issues/9787). Would this be a cause for the mentioned behaviour?
Hi @Phillip Whelan, I have updated the fluent-bit to version 4.0.7 and enabled the SQLDB as well. The deployment ran without an issue for about 1 day and now I'm still having the same issue described above. Would you be able to help? Please find the logs attached below.
Copy code
[2025/09/04 03:45:15] [debug] [input:kubernetes_events:kubernetes_events.0] inserted k8s event: uid=bdffcd4e-6847-410f-9042-4f52beaa6569, resource_version=1081785301, last=1756957515000000000
[2025/09/04 03:45:15] [debug] [input:kubernetes_events:kubernetes_events.0] is_filtered: uid=5c0d231b-2d40-477f-be82-3613d1f07a6f exists=1
[2025/09/04 03:45:16] [debug] [input:kubernetes_events:kubernetes_events.0] is_filtered: uid=c7ea5b8d-dbb0-4416-9edc-5a1fdd1d6525 exists=1
[2025/09/04 03:45:16] [debug] [input:kubernetes_events:kubernetes_events.0] is_filtered: uid=aaf290d8-1530-444f-a2cc-95c98f686eb4 exists=1
[2025/09/04 03:45:16] [debug] [input:kubernetes_events:kubernetes_events.0] could not process payload, incomplete or bad formed JSON: af68
After this
incomplete or bad formed JSON
, I continuously get the same log and fluent-bit no longer proceeds with events collection.
p
I would raise an issue @Akila Induranga with all the details and ideally a reproducer or captured partial messages that trigger the issue
Whoops, didn't mean to post to the channel too
a
Thanks for the reply Pat, I will raise an issue on GitHub.