Hi, does anyone know what is the root cause for th...
# fluent-bit
m
Hi, does anyone know what is the root cause for the error that i get while running fluent-bit as a sidecar inside a kubernetes pod:
Copy code
[2025/09/18 07:33:47] [ info] [input:tail:tail.0] initializing
[2025/09/18 07:33:47] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/09/18 07:33:47] [error] [/src/fluent-bit/plugins/in_tail/tail_fs_inotify.c:360 errno=24] Too many open files
[2025/09/18 07:33:47] [error] failed initialize input tail.0
[2025/09/18 07:33:47] [error] [engine] input initialization failed
This gets fixed by increasing ulimit , but i dont want to do that. Is there any other way to fix this , like limiting the no of files opened by fluent-bit. I am also attaching the snippet from configmap of fluent-bit.
Copy code
input.conf: |
      [INPUT]
          Name              tail
          Path              /var/log/container/*.log 
          Parser            docker
          Tag               container
          Refresh_Interval  5
          Read_from_Head    True
          Skip_Long_Lines   On

filter.conf: |
   
    [FILTER]
        Name                parser
        Match               container
        Key_Name            log
        Parser              docker
        Reserve_Data        True

    [FILTER]
        Name              lua
        Match             container
        script            /fluent-bit/etc/parser.lua
        call              parse_log
p
That's a fundamental restriction, you have X amount of files (note these are file descriptors so can be sockets/etc. too) that you are allowed to monitor by the kernel.
You can restrict the number of files by changing your
path
value to reduce it
m
can we also use
Exclude_Path
and
Ignore_Older
in order to restrict the number of files? and will this log collection in some way?
p
yes and of course it will affect log collection if you ignore log files or are unable to monitor them
what's the big issue with ulimit?
you should also use a multiline parser for your log files - there's no guarantee from the container runtime around when it will split a single log message into multiple lines, the docs provide recommended options to use: https://docs.fluentbit.io/manual/data-pipeline/inputs/tail#multiline-and-containers
plus you have
read_from_head
set so it will always read the entire log file every time it starts which may not be what you want, e.g. a pod is restarted
n
"[2025/09/18 073347] [error] [/src/fluent-bit/plugins/in_tail/tail_fs_inotify.c:360]" Sometimes this error appears, but on many VMs Fluent Bit runs without any issue. The behavior is inconsistent — it works fine on some machines, while occasionally showing this error on others. Question: How can I properly test if Fluent Bit is working correctly in such cases?
Copy code
fluent-bit.conf: |
    [SERVICE]
        Flush        1  
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

    @INCLUDE input.conf
    @INCLUDE filter.conf
    @INCLUDE output.conf

  input.conf: |
      [INPUT]
          Name              tail
          Path              /var/log/object-storage/*.log 
          Parser            docker
          Tag               object-storage
          Refresh_Interval  5
          Read_from_Head    True
          Skip_Long_Lines   On
@Pat
@Anurag Gupta
a
Try turning inotify off
otherwise you will need to adjust file handlers