Hello team, I met a problem that the fluent-bit ca...
# fluent-bit
r
Hello team, I met a problem that the fluent-bit cannot grab the existing log. The scenario is that I have some logs for an application in the cluster that are collected as files in the fluent-bit pod, then I add an INPUT
tail
below:
Copy code
[INPUT]
        Name              tail
        Tag               test
        Path              /var/log/containers/test*_test*.log
        Multiline.parser  docker, cri, go
        Docker_Mode       true
        Docker_Mode_Flush 4
        Mem_Buf_Limit     30MB
        Skip_Long_Lines   true
        Refresh_Interval  5
        Buffer_Chunk_Size 320KB
        Buffer_Max_Size   768KB
By this fluent-bit can only tail from the newly created logs while skipping the existing logs, as you can see I didn't set the
Ignore_Old
option. I tried to add the
Read_from_Head  true
and fluent-bit does grab the existing old logs, but here comes another issue when the fluent-bit config is updated or the fluent-bit pod is restarted, the log file will be tailed again from the beginning and leads to duplication. Do you have any idea how to solve this?
p
Turn on debug logs and check what it reports, more often than not it'll be dismissing the logs for some reason or potentially unable to access them due to file permissions
Also what version are you on and make sure it's the latest?
The docker mode options are legacy ones and I think even conflict with multi line support. Remove those.
Use a db file is the way for persistence btw, this is likely what you need. The helm chart defaults to that config for example
Ah my bad, it doesn't but it should lol Anyway, db file is what you want to persist where you are up to