hi team, We are facing data loss when fluent bit is restarted - the logs produced between stopping and starting of the fluent bit agent are lost, we noticed that the db entry gets deleted and recreated every time. Please find the details below Host : amazon linux 2 Architecture : x86_64 Fluent bit version : 3.2.10 Input config :

[INPUT]
    Name tail
    Path /var/log/cloud-init.log
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  cloud-init.log
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /var/log/messages
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  messages
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /var/log/secure
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  secure
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /var/log/yum.log
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  yum.log
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /root/.newrelic/newrelic-cli.log
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  newrelic-cli.log
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

Before Agent stops (fb.db):



sqlite> select * from in_tail_files where name = '/var/log/test.log';
id|name|offset|inode|created|rotated
33|/var/log/test.log|85|12713160|1762270005|0
After Infrastructure Agent starts (fb.db):


After Agent starts 

sqlite> select * from in_tail_files where name = '/var/log/test.log';
id|name|offset|inode|created|rotated
41|/var/log/test.log|114|12713160|1762271396|0

Please notice the change in created at time stamp We tried the same on other Operating Systems and architectures but they all seem to be working fine.

3.2.10 is unsupported now so I'd start with stepping up: https://github.com/fluent/fluent-bit/security/policy

@Pat, sorry I missed the test.log input block

[INPUT]
    Name tail
    Path /var/log/test.log
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  test.log
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /var/log/cloud-init.log
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  cloud-init.log
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /var/log/messages
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  messages
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /var/log/secure
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  secure
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

[INPUT]
    Name tail
    Path /var/log/yum.log
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Path_Key filePath
    Tag  yum.log
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

yeah there's a fair bit of change since that release: https://github.com/fluent/fluent-bit/compare/v4.2.0...v3.2.10