Hello! I am testing out fluent-bit, and wanted to ...
# fluent-bit
a
Hello! I am testing out fluent-bit, and wanted to ask for some help/suggestions. I need to move logs from one fluent-bit instance to another through files (shuffled through a network barrier by Apache nifi). Before (using rsyslog) this was done using minute files that were created and moved. I've been looking at using the file output/input, but it does not look obvious if I can make minute/timestamped output files or read-and-discard input file. I could ofc use exec for the input part, but there is no exec output?
p
Using files is not a good idea if you need ordering maintained
I did a proof of concept/silly idea years ago to show fluent bit as rsync lol
Order of input will not be guaranteed to be maintained for output though is my main point so if this is required I would not do it
An easy way to write whatever files you need though would be as a side effect from the lua filter and just use a null output
a
I would assume the timestamp in the filenames would make the order stay
p
Potentially, order within the files cannot be guaranteed but that may be ok for you
a
Considering using logstash as a collector now, just using fb on the endpoints 🤔