Hello all, I am using fluentbit tail input plugin with the file output plugin to ship log files, I monitored the process memory usage and noticed that memory does not decrease after shipping all the files even though I added mem_buf_limit parameter (please note that I left the process running for hours to see if there is any memory drop but nothing happened). I dont know if this is the expected behavior or if there is any other way to control the memory used by the process. Can you please assist :) ? this is the configuration used with the memory graph:

pipeline:
 inputs:
  - name: tail
   path: path/to/files/*, path/to/files/*/*, path/to/files/*/*/*
   exclude_path: '*.log.0,*.log.1,*.log.2,*.log.3,*.log.4,*.log.5
   tag_regex: (path/to/files/*|path/to/files/*/*|path/to/files/*/*/*)(?<original_path>.*)
   tag: tail:<original_path>
   read_from_head: true
   skip_long_lines: true
   ignore_older: 1d
   db: offsets.db
   db.locking: true
   inotify_watcher: false
   mem_buf_limit: 256MB
 filters:
  - name: lua
   match: tail:*
   script: compute_target_path.lua
   call: compute_target_path

  - name: rewrite_tag
   match: tail:*
   rule: $target_path ^(.*)$ $1 false

 outputs:
  - name: file
   match: '*'
   path: output/path
   mkdir: true
   format: template
   template: '{log}'

Unrelated but there's no guarantee on ordering from input to output so your lines in output file may be different order to the input one. Unlikely but possible. You are also mixing

read_from_head

with

db

which is not correct.

db

behaves as

read_from_head

the first time it sees the file, otherwise it starts from wherever it got up to in the file. Why are you just writing from one file to another, and just including the same lines in each with no extra data? It looks like all you do is adjust the output path but feels like there are easier ways to do that...

Thanks for your response and for the tips. The idea is just to investigate the memory issue with a simpler version (tail -> file) instead of using an s3 instance (we had the same memory issue using s3 output). Do we expect the memory to never drop even after leaving it for hours ?

What version are you on and if it's not the latest try that?

I did another test with fluent bit 4.2.0 but had the same memory issue. So, I'm wondering if the binary will just keep it for chunks to reuse as you said @Pat. @lecaros can you please confirm this behavior ? if yes how can we prevent against it since it might lead to OOM?

is that a controlled environment? (no other processes writing files, no rotation, etc.) Checking metrics is a good idea, verify the chunks' ones, whether they are dropping or just keep growing. If you have a lot of files there, combined with the ignore_older 1d, it will keep at least 32kb for each file until it's discarded by it's modify age. Is the same issue without the filters?