hi team, Im getting this log in fluentbit which is deployed in production, can anyone help me in understanding why this happens and what is the recommended config if we get these logs

[2025/10/23 09:39:52] [ warn] [input:tail:tail.0] purged rotated file while data ingestion is paused, consider increasing rotate_wait
[2025/10/23 09:41:31] [ info] [input] tail.0 resume (mem buf overlimit)
[2025/10/23 09:41:31] [ warn] [input] tail.0 paused (mem buf overlimit)
[2025/10/23 09:41:31] [ info] [input] pausing tail.0
[2025/10/23 09:41:31] [ info] [input] resume tail.0
[2025/10/23 09:41:31] [ info] [input] tail.0 resume (mem buf overlimit)

currentconfig:

[SERVICE]
    Flush        30
    Grace        5
    Daemon       Off
    Log_Level    info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc-operator/custom-parsers.conf
    Coro_Stack_Size    24576
    HTTP_Server  On
    Listen 0.0.0.0
    HTTP_Port    2020
    storage.path  /buffers
[INPUT]
    Name         tail
    Buffer_Chunk_Size  100MB
    Buffer_Max_Size  800MB
    DB  /tail-db/tail-containers-state.db
    DB.locking  true
    Ignore_Older  1h
    Mem_Buf_Limit  1000MB
    Parser  cri-log-key
    Path  /var/log/containers/*.log
    Refresh_Interval  5
    Skip_Long_Lines  On
    Tag  kubernetes.*
    storage.pause_on_chunks_overlimit  on

Backpressure - your output is not keeping up with the input

hey @Pat thanks for the reply, what is the recommended config and what needs to be changed

If your output is not keeping up then you have a level of buffering (e.g. for spikes but not for just being too slow all the time) but if that is exceeded then you need to choose: 1. Stop ingesting data (which is what you see here) 2. Throw away data

wont it go OOM killed if we increase buffer

Otherwise you need more resources either to run FB to send to the output or your backend that is receiving the data needs more resources to cope with more output

but is there any way other thn increasing e resources either to run FB, like making storage as filesystem

I've no idea what you have configured in your infra, if you need more memory then you need more memory so higher limits. That is only if it is a transient spike though: you need to figure that out - is it just unable to cope with the data rates required or is there a spike you need to cope with?

when there is a spike we start seeing this issue

So it just needs more buffering to cope with the spike? You can test that by temporarily increasing things to confirm.

sure, also what if spike is transient spike just curious

Filesystem buffering is possible but that ultimately can fill up too so that's why I want you to confirm if it is transient or not. Otherwise you will just fill up the filesystem then see the same thing

what if it just unable to cope with the data rates; for this usecase Filesystem buffering will be helpful ?

No, you just have more buffering but ultimately that will fill up too. Hence my question to you about confirming is it a spike or just constant underperformance.

will check 1 min

sounds like your output cannot cope for whatever reason - you can check why by looking at your infra and FB

its running there no crashloop as of now, I mean instead of increasing resource I was thinking of optimizinf cinfigs of fluentbit for a long term fix

That's not what I asked, what do the metrics/logs say?

image.png

Nice pics, what does it show?

kubectl top pod k8s-prod-logging-zone-1b-fluentbit-wlspb -n logs NAME CPU(cores) MEMORY(bytes) k8s-prod-logging-zone-1b-fluentbit-wlspb 2m 7Mi

again, are you getting errors/retries and is your CPU saturated?

there is a difference in input and output and output retried metric has increased, No cpu is not saturated current usage is 2m out of cpu: 352m

so you can check why it is being retried and see if it's an issue with the backend you are sending too

workers for output ? we are using elasticsearch, fluentbit forwards to elasticsearch

check the docs to see how to use it, there's probably a default number of workers configured but not sure what that is

ok thanks for the info, so moving to filesystem wont solve this as it can make that full too

sure, it's just another buffer