Hi we do have another question
Does FluentBit tail input support a high throughput logs generation?
Indeed we do have a pod that generates ~1M logs in 2 min (1 log/client request) and we do observe missing logs collected by the FluentBit DeamonSet for container logs.
I tried to run a test with https://github.com/kubernetes/kubernetes/tree/v1.6.13/test/images/logs-generator and got the issue at similar pace
• 100K in 3 min with the generator --> ok
• 500K in 3 min --> ko
• 1M in 3 min --> ko
Can FluentBit miss the inotify events?
u
Ujjwal Sharma
09/17/2025, 3:16 PM
I have seen log losses when the file rotates at a high log volume and fluentbit will not be able to find the file to tail anymore. You can tweak your configuration to support higher load. We easily support 25k TPS with our configuration
h
Hung
09/17/2025, 3:16 PM
hey thanks @Ujjwal Sharma for the feedback
Hung
09/17/2025, 3:17 PM
when you say "tweak your config" .. you talking about the input tail settings?
Hung
09/17/2025, 3:18 PM
do you have an example of settings for such high load?
p
Pat
09/17/2025, 4:07 PM
It'll depend on your specific set up a bit but generally the answer is ensuring Fluent Bit has enough resources to keep up with your data rates.
You'll need to make sure the outputs can cope too, Fluent Bit will send to them and if they back up then it'll have to.
Ultimately it's a general resource problem for a process, it needs enough to keep up with the events it's receiving.
👍 1
h
Hung
09/17/2025, 4:20 PM
The output rate is much much lower than the Redpanda could take so I am not very worried about it.
Hung
09/17/2025, 4:39 PM
the fluentbit pod resources consumption is much lower than the limits too