Hey folks I am facing an issue We started to see drop in log Fluent Community #fluent-bit

Hey folks, I’am facing an issue We started to see ...

Sujay

10/15/2025, 12:35 AM

Hey folks, I’am facing an issue We started to see drop in logs when there is high traffic when we started to investigate got to know that our log files in /var/log/container is being rotated very frequently by kubelet as kubelet has a policy to rotate log files if it reaches 10Mi, so I’m suspecting when fluentbit is currently reading the file its getting rotated and log is not being forwarded, is there any plugin or config to tackle the issue at scale to read the logs before it gets rotated

Pat

10/15/2025, 8:22 AM

If you mean Fluent Bit then it should still follow the rotated log inode

Sujay

10/15/2025, 8:24 AM

does it also read the file when its in .gz format when its rotated

Pat

10/15/2025, 8:26 AM

No but that's a new file and not the original inode. Potentially you also have back pressure and other issues slowing things down. Are you sure it's down to the input missing it or could it be output not keeping up? You can check the metrics to see what's happening

Sujay

10/15/2025, 11:00 AM

https://github.com/fluent/fluent-bit/issues/2110

Sujay

10/15/2025, 11:00 AM

You can refer the above issue, we are facing same logs get dropped at scale

Sujay

10/15/2025, 11:14 AM

How would I make fluentbit reads faster when write are too high

Pat

10/15/2025, 11:59 AM

I don't understand your last question, ultimately if your output cannot keep up then you are buggered 🙂 It looks like @lecaros was looking at the issue so may have more info

Sujay

10/15/2025, 1:08 PM

I see other peeps have also faces similar issue i got few threads in this slack, but resolution is unknown

Sujay

10/15/2025, 1:09 PM

example: https://fluent-all.slack.com/archives/C2RFHMHJB/p1756103200244559

lecaros

10/15/2025, 1:38 PM

hello there. Do you have a consistent repro we can use to troubleshoot? what's your config? what's in the log file? how are the metrics? do you see backpressure? (that example of similar issue looks different to me, but I haven't read it in full)

Sujay

10/22/2025, 3:36 AM

Hey @lecaros, Actually this happened in our prod where we observed that few logs were missing unfortunately I don’t have a repro, I saw your comment on same thread, so we can implement refresh_interval to solve this ? Will this help fluentbit to read at same speed as write?

Sujay

10/22/2025, 3:38 AM

Also I saw mention about disk, so how does fluentbit reads, does it stores in memory and thn sends to target like (loki or elastic search) or it first stores the log on the disk and read it in our configuration We have not mentioned storage_type explicitly we are using default which is memory, will setting storage_type to filesystem help ?

Sujay

10/22/2025, 4:46 AM

Copy code

[2025/10/22 04:24:56] [ info] [input:tail:tail.0] inotify_fs_add(): inode=159640112 watch_fd=20638 name=/var/log/containers/gamma-log-loadtest-2_fl-dep_writer-212ad23fa11f375b09d587b06297f3c72336af5def664332bed8431ab916ad01.log
[2025/10/22 04:24:57] [ warn] [input] tail.0 paused (mem buf overlimit)
[2025/10/22 04:24:57] [ info] [input] pausing tail.0
[2025/10/22 04:25:02] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=159651359 watch_fd=20637
[2025/10/22 04:25:02] [ info] [input] resume tail.0
[2025/10/22 04:25:02] [ info] [input] tail.0 resume (mem buf overlimit)
[2025/10/22 04:25:03] [ warn] [input] tail.0 paused (mem buf overlimit)
[2025/10/22 04:25:03] [ info] [input] pausing tail.0
[2025/10/22 04:25:03] [ info] [input] resume tail.0
[2025/10/22 04:25:03] [ info] [input] tail.0 resume (mem buf overlimit)




[2025/10/22 04:25:13] [ warn] [input] tail.0 paused (mem buf overlimit)
[2025/10/22 04:25:13] [ info] [input] pausing tail.0
[2025/10/22 04:25:13] [ warn] [input:tail:tail.0] purged rotated file while data ingestion is paused, consider increasing rotate_wait

[2025/10/22 04:41:11] [ info] [input] resume tail.0
[2025/10/22 04:41:11] [ info] [input] tail.0 resume (mem buf overlimit)
[2025/10/22 04:41:13] [error] [/src/fluent-bit/plugins/in_tail/tail_file.c:1655 errno=2] No such file or directory
[2025/10/22 04:41:13] [error] [/src/fluent-bit/plugins/in_tail/tail_fs_inotify.c:147 errno=2] No such file or directory
[2025/10/22 04:41:13] [error] [input:tail:tail.0] inode=2061578 cannot register file /var/log/containers/log-writer-loadtest_fl-dep_writer-bf61e5050526a6f539bc8ba9070560278bb3d285f7b945bb0c8e6ca73fd218d5.log

1. I tried pushing logs with message size of 250kb 2. Generate exactly 500,000 lines at maximum speed (no time limit) 3. Pause for exactly 5 minutes 4. Repeat forever and above is the log of fluentbit pod and below is the current config

Copy code

[SERVICE]
    Flush        30
    Grace        5
    Daemon       Off
    Log_Level    info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc-operator/custom-parsers.conf
    Coro_Stack_Size    24576
    HTTP_Server  On
    Listen 0.0.0.0
    HTTP_Port    2020
    storage.path  /buffers
[INPUT]
    Name         tail
    Buffer_Chunk_Size  100MB
    Buffer_Max_Size  800MB
    DB  /tail-db/tail-containers-state.db
    DB.locking  true
    Ignore_Older  1h
    Mem_Buf_Limit  1000MB
    Parser  cri-log-key
    Path  /var/log/containers/*.log
    Refresh_Interval  5
    Skip_Long_Lines  On
    Tag  kubernetes.*
    storage.pause_on_chunks_overlimit  on
    
[FILTER]
    Name        kubernetes
    Buffer_Size  0
    K8S-Logging.Exclude  On
    Kube_CA_File  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Tag_Prefix  kubernetes.var.log.containers
    Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Token_TTL  600
    Kube_URL  <https://kubernetes.default.svc:443>
    Match  kubernetes.*
    Merge_Log  On
    Use_Kubelet  Off
    namespace_labels  On

Sujay

10/22/2025, 4:48 AM

@lecaros @Pat what would you recommend would you please give your suggestion on this

Open in Slack

Previous Next