Thanks for the links! I went through the workshop labs - very helpful for understanding the backpressure mechanics. We're planning to switch to storage.type = filesystem eventually, but we'd really like to understand the root cause first. We're still seeing data loss and want to figure out what's going on. Here's our situation: - storage.type = memory (default) - Retry_Limit = False (unlimited retries) - db is enabled on all tail inputs - No OOM kills observed - We see paused/resumed (mem buf overlimit) cycles - The lost data is lines from the middle of files, not the tail end - threaded: true on the tail input. Our input config: - name: tail read_from_head: true refresh_interval: 5 inotify_watcher: false skip_empty_lines: false skip_long_lines: false mem_buf_limit: 30Mb buffer_chunk_size: 32Kb buffer_max_size: 2Mb threaded: true static_batch_size: 5Mb event_batch_size: 5Mb db: /fluent-bit/data/test.db Our output config: - name: tcp match: test format: msgpack workers: 1 retry_limit: false net.keepalive: true net.keepalive_idle_timeout: 3600 Are there any other known edge cases in the memory storage engine where chunks can be silently dropped without an OOM event?

Some context on our file management: we have a script that periodically copies log files into the directory that tail monitors, and also cleans up old files from that directory. However, we only validate data integrity for files that still exist at the time of checking - we don't count deleted files as data loss. The losses we're seeing are missing lines from files that are present and fully readable on disk.