Jay Mahendran
09/18/2025, 7:39 PMread_from_head: true + storage.type: filesystem for FluentBit 4.0.9, expecting that on restart, Fluent Bit resumes reading from the last offset. But what I am seeing is that the entire file gets re-sent on every restart, even though:
• The DB file is persistent
• The DB path and file path are stable
Here’s a simplified config:
service:
daemon: off
flush: 1
health_check: On
hot_reload: On
http_listen: 0.0.0.0
http_port: 2020
http_server: On
log_level: info
parsers_file: /fluent-bit/etc/fluentbit-parser.conf
scheduler.base: 1
scheduler.cap: 120
storage.checksum: On
storage.delete_irrecoverable_chunks: On
storage.max_chunks_up: 128 # 256 MB in-memory buffer
storage.path: /data/fluent-bit
storage.sync: full
pipeline:
inputs:
- name: tail
alias: tail.tests-log
DB.journal_mode: WAL
DB.locking: true
DB: /data/fluent-bit/fluent-bit-tests.db
parser: json-log-parser
path_key: logfile
path: /var/log/tests/compacted_tests.ndjson
read_from_head: true
storage.type: filesystem
buffer_max_size: 1M
buffer_chunk_size: 1MB
tag: tests
outputs:
- name: stdout
match: "*"
Is this expected behavior? Or is something wrong in how I am configuring things? Appreciate any insights 🙏emmacz
09/18/2025, 10:56 PMread_from_head: true ? It will skip 'last offset' and always read from whole file from head.Jay Mahendran
09/18/2025, 10:58 PMread_from_head: true because I want Fluent Bit to read from the beginning of the file on the very first run, but still expected it to track the offset afterward using the DB file and avoid re-sending on restart.
So if I understand correctly, you're saying that even with a persistent DB, if read_from_head: true is set, Fluent Bit will always start from the beginning on every restart?
If so — does that mean the read_from_head setting takes precedence over DB tracking, even when the DB exists and is valid?Pat
09/19/2025, 10:04 AMJay Mahendran
09/19/2025, 3:48 PMread_from_head flag only the new log lines gets forwarded for a new file.Then Fluent Bit only forwards new lines written after it starts — it does not process the existing content already in the file.
That’s why I initially used read_from_head: true, to make sure it picks up all content from new files.
But I didn’t realize that would also force it to ignore the DB offsets on restarts.
Is there a recommended way to get:
• Full file read on first encounter, and
• Offset tracking for subsequent runs,
• without triggering re-reads on every restart?Anurag Gupta
09/22/2025, 3:37 AMAnurag Gupta
09/22/2025, 3:37 AMAnurag Gupta
09/22/2025, 3:38 AMJay Mahendran
09/22/2025, 5:25 PMAnurag Gupta
09/22/2025, 6:07 PMPat
09/23/2025, 11:16 AMPat
09/23/2025, 11:17 AMPat
09/23/2025, 11:19 AMPat
09/23/2025, 11:24 AM