Hi Team, With the following Fluent Bit configuration, I'm aiming to process approximately 600K events per minute. These events are distributed across 40 files in NFS directory, which are generated hourly. Additionally, some events may still be written to older files. To optimise performance, filesystem storage has been disabled. Issue Observed • Events that were already ingested in real-time are being re-ingested on an hourly basis. Impact • This results in duplicate events and delayed ingestion, affecting data accuracy and timeliness. Could you please review the configuration and let me know if there are any misconfigurations or improvements that can help prevent re-ingestion and ensure consistent performance?

[SERVICE]
    Flush                       1
    Grace                       15
    Daemon                      Off
    HTTP_Server                 On
    HTTP_Listen                 0.0.0.0
    HTTP_Port                   2020
    Health_Check                On
    Storage.metrics             On
    Storage.sync                Normal
    Storage.checksum            Off
    Storage.path                /data/fluentbit/storage
    Storage.backlog.mem_limit   24576M
    Storage.backlog.chunk_size  1M
    Storage.max_chunks_up       128
    Storage.delete_oldest       On
    storage.delete_irrecoverable_chunks On
    storage.pause_on_chunks_overlimit On
    storage.backlog.flush_on_shutdown On
    Hot_Reload                  On
    Log_Level                   Info
    Log_File                    /mnt/prod/fluentbit/logs/log-2025-10-23.log
    Workers                     12

[INPUT]
    Name                        tail
    Path                        /logs/source/*.ndjson
    Path_Key                    filepath
    Tag                         daily.logs
    DB                          /data/fluentbit/db/fluentbit.db
    DB.Sync                     full
    DB.compare_filename         true
    DB.locking                  true
    Mem_Buf_Limit               24576M
    Buffer_Chunk_Size           1M
    Buffer_Max_Size             4M
    Skip_Long_Lines             On
    Skip_Empty_Lines            On
    Refresh_Interval            2
    Ignore_Older                24h
    Rotate_Wait                 86400
    Read_from_Head              true
    Inotify_Watcher             true
    Exit_On_Eof                 Off
    Multiline                   Off
    threaded                    true
    File_Cache_Advise           Off

[OUTPUT]
    Name                        http
    Match                       daily.logs
    Host                        clickhouse-chproxy.clickhouse-cluster.svc.cluster.local
    Port                        8080
    URI                         /?query=INSERT%20INTO%20events_distributed
    Format                      json
    json_date_key               timestamp
    json_date_format            iso8601
    Header                      Content-Type application/json
    Workers                     12
    Retry_Limit                 no_retries
    net.connect_timeout         20
    net.keepalive               On
    net.keepalive_idle_timeout  300
    net.keepalive_max_recycle   2000
    net.max_worker_connections  0
    storage.total_limit_size    8GB
    Compress                    gzip

cc: @Pat, @lecaros

You have read from head on so if things restart you'll reingest already handled logs

Hi @Pat thank you so much for prompt response! DB is configured, will this not handle? Pod is not restarted, please find attached screenshots.

You should not configure both

Could enabling worker threads be causing duplicate ingestion?

Do you mean path duplication? Yeah if you're matching the rotated file as well then it'll be ingested again

Log files are not move to different location till 24 hours then we remove those files to different directory

I don't understand - do you rotate them or just leave them there? If you don't rotate them then that's fine, it won't read them again (if you turn read from head off)

just leave the file, no rotation