FuentBit Tail Input I am using DB tracking + `rea...
# fluent-bit
j
FuentBit Tail Input I am using DB tracking +
read_from_head: true
+
storage.type: filesystem
for FluentBit 4.0.9, expecting that on restart, Fluent Bit resumes reading from the last offset. But what I am seeing is that the entire file gets re-sent on every restart, even though: • The DB file is persistent • The DB path and file path are stable Here’s a simplified config:
Copy code
service:
  daemon: off
  flush: 1
  health_check: On
  hot_reload: On
  http_listen: 0.0.0.0
  http_port: 2020
  http_server: On
  log_level: info
  parsers_file: /fluent-bit/etc/fluentbit-parser.conf
  scheduler.base: 1
  scheduler.cap: 120
  storage.checksum: On
  storage.delete_irrecoverable_chunks: On
  storage.max_chunks_up: 128 # 256 MB in-memory buffer
  storage.path: /data/fluent-bit
  storage.sync: full

pipeline:
  inputs:
    - name: tail
      alias: tail.tests-log
      DB.journal_mode: WAL
      DB.locking: true
      DB: /data/fluent-bit/fluent-bit-tests.db
      parser: json-log-parser
      path_key: logfile
      path: /var/log/tests/compacted_tests.ndjson
      read_from_head: true
      storage.type: filesystem
      buffer_max_size: 1M
      buffer_chunk_size: 1MB
      tag: tests

  outputs:
    - name: stdout
      match: "*"
Is this expected behavior? Or is something wrong in how I am configuring things? Appreciate any insights 🙏
👀 1
e
@Jay Mahendran any reason why you have
read_from_head: true
? It will skip 'last offset' and always read from whole file from head.
j
@emmacz I originally set
read_from_head: true
because I want Fluent Bit to read from the beginning of the file on the very first run, but still expected it to track the offset afterward using the
DB
file and avoid re-sending on restart. So if I understand correctly, you're saying that even with a persistent DB, if
read_from_head: true
is set, Fluent Bit will always start from the beginning on every restart? If so — does that mean the
read_from_head
setting takes precedence over DB tracking, even when the DB exists and is valid?
p
You do not need to set read from head, if there is no entry in the db then that is the same as offset 0 so it should read from the start
j
@Pat the behavior that I am observing is if I don't set the
read_from_head
flag only the new log lines gets forwarded for a new file.Then Fluent Bit only forwards new lines written after it starts — it does not process the existing content already in the file. That’s why I initially used
read_from_head: true
, to make sure it picks up all content from new files. But I didn’t realize that would also force it to ignore the DB offsets on restarts. Is there a recommended way to get: • Full file read on first encounter, and • Offset tracking for subsequent runs, • without triggering re-reads on every restart?
a
hmm I don't think this is intended behavior, it should read from head once and then resume from the latest offset
I only know that cause in my tests I generally have to remove the DB parameter to make sure it grabs the latest 🙂
Are you seeing any errors in the log around the DB file
j
No other errors in the logs, all I could see in the logs was upon restart the offset in the DB would get updated to the latest byte size of the log file tailed, without forwarding any data. I had multiple tail inputs sharing a single DB file for offset tracking. Fixed it by assigning individual DB files to each tail input. Offset tracking now works correctly with separate databases. Is it expected to have individual DBs for each tail input? There's no mention of this requirement in the official documentation
a
Oh that makes sense, glad it was fixed probably a good update to add to docs.
p
yeah you can't share the DB file so that's why, I'm sure it says that somewhere....
the docs do cover the behaviour of read-from-head plus db here: https://docs.fluentbit.io/manual/data-pipeline/inputs/tail#database-file However I cannot see an explicit, do not share it anywhere so will add
You may be able to share a db file as long as there is no overlaps in the inodes that db file is then tracking but this is impossible to guarantee with multiple plugins, e.g. an inode for a file may be reused for another, so just separating them is probably best.