Hey 🤠 We have a large fleet of servers that is running fluent-bit as a docker container, with systemd input. Whenever we push out changes to the configuration of fluent-bit, ansible will replace the container with a new one. This generates a lot of systemd logs that fluent-bit is sending to opensearch. The massive influx of systemd logs seems like maybe fluent-bit is resending already processed logs. This is how we have configured the pipeline for systemd logs:

---
pipeline:
  inputs:
    - name: systemd
      tag: systemd.*
      read_from_tail: on
      threaded: true
      lowercase: on
      db: /fluent-bit/db/systemd.db
      storage.type: memory # Filesystem buffering is not needed for tail input since the files are stored locally.
      mem_buf_limit: 250M
      alias: in_systemd

We have set

db

as well as

read_from_tail: on

so our thoughts were that the fluent-bit container should not resend already processed logs, is this true?