Hello I'm using `storage.delete_irrecoverable_chun...
# fluent-bit
k
Hello I'm using
storage.delete_irrecoverable_chunks On
option but
.flb
files are not deleted automatically. How to solve this problem?
Copy code
// fluent-bit.conf

[SERVICE]
    Flush 1
    parsers_file parsers.conf
    Daemon Off
    Log_Level info
    storage.path .\storage
    storage.max_chunks_up 128
    storage.sync full
    storage.checksum On
    storage.delete_irrecoverable_chunks On
I got the error above.
Copy code
[2025/05/22 14:48:05] [error] [storage] format check failed: tail.0/51904-1746111458.203209800.flb
[2025/05/22 14:48:05] [ info] [storage] [cio file] invalid crc32 at 5200-1745827407.145630700.flb/.\storage\tail.0\5200-1745827407.145630700.flb
But
51904-1746111458.203209800.flb
file is not deleted automatically.
My expectation is when I run the fluent-bit, fluent-bit will delete the file which is invalid.
p
What version are you using?
What to the logs say?
Looking at the code it should log an error saying it is discarding it: https://github.com/fluent/fluent-bit/blob/7859e8daf574f2ee3556ddaf0a067d59076f2707/lib/chunkio/src/cio_scan.c#L107-L114 Note this is specifically if it is enabled, the chunk is corrupted and the error is file size or bad layout. I'm not sure if format errors falls into that @Leonardo Almiñana?
k
My version: 4.0.0 Let me correct my question.
.flb
files are automatically deleted in general situation. But if fluent-bit is terminated unexpectedly, some
.flb
files are corrupted. In this case, when I start the fluent-bit again, fluent-bit is paniced. Here is the corrupted file. Is there anyway to skip corrupted file when I start the fluent-bit?
l
What do you mean by panic? Could you share the entire log file?
As @Pat said the chunk is only discarded if the critical structure is corrupted, data corruption is not included in this category
Also, have you been able to reliably reproduce the chunk corruption scenario? If so I'd like to look at it to see if it's something that can be remediated
Although if you are killing fluent-bit mid write it might not be possible
k
there's no error log but fluent-bit is terminated after the log below. fluent-bit is terminated when I put the corrupted file to
tail.0
directory.
Copy code
[2025/05/23 18:03:22] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2025/05/23 18:03:22] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2025/05/23 18:03:22] [debug] [storage_backlog:storage_backlog.1] created event channels: read=1904 write=1908
[2025/05/23 18:03:22] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 95.4M
[2025/05/23 18:03:22] [debug] [filter:modify:modify.2] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [filter:modify:modify.6] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [filter:modify:modify.7] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [filter:modify:modify.8] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [opensearch:opensearch.0] created event channels: read=2064 write=2068
[2025/05/23 18:03:22] [debug] [tls] successfully loaded certificates from windows system store.
[2025/05/23 18:03:22] [debug] [output:opensearch:opensearch.0] host=search-eclipse-log-plw4vsgtpgmd73abzarvfopwla.ap-northeast-2.es.amazonaws.com port=443 uri=/_bulk index=fluent-bit type=_doc
[2025/05/23 18:03:22] [debug] [router] match rule tail.0:opensearch.0
[2025/05/23 18:03:22] [debug] [router] match rule storage_backlog.1:opensearch.0
[2025/05/23 18:03:22] [ info] [sp] stream processor started
l
Could you share your configuration with me?
And the entire log file
You can send them to me in private if you'd prefer to do so
I'll reproduce the issue and get back to you
k
Im not in the office now so I’ll give you the entire log and configuration soon. Thank you for your reply.
l
Great, just ping me when you can and we'll sort it out
😀 1
k
Copy code
OS: Window
Version: 4.0.0
Version for git commit: 712b48d7c387c87c922075aa5556afe648e36c4b
Full log when I execute
fluent-bit.exe -c fluent-bit.conf
Copy code
[2025/05/24 09:55:01] [ info] Configuration:
[2025/05/24 09:55:01] [ info]  flush time     | 1.000000 seconds
[2025/05/24 09:55:01] [ info]  grace          | 5 seconds
[2025/05/24 09:55:01] [ info]  daemon         | 0
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info]  inputs:
[2025/05/24 09:55:01] [ info]      tail
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info]  filters:
[2025/05/24 09:55:01] [ info]      parser.0
[2025/05/24 09:55:01] [ info]      parser.1
[2025/05/24 09:55:01] [ info]      modify.2
[2025/05/24 09:55:01] [ info]      parser.3
[2025/05/24 09:55:01] [ info]      parser.4
[2025/05/24 09:55:01] [ info]      lua.5
[2025/05/24 09:55:01] [ info]      modify.6
[2025/05/24 09:55:01] [ info]      modify.7
[2025/05/24 09:55:01] [ info]      modify.8
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info]  outputs:
[2025/05/24 09:55:01] [ info]      opensearch.0
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info]  collectors:
[2025/05/24 09:55:01] [ info] [fluent bit] version=4.0.0, commit=712b48d7c3, pid=31780
[2025/05/24 09:55:01] [debug] [engine] coroutine stack size: 98302 bytes (96.0K)
[2025/05/24 09:55:01] [ info] [storage] ver=1.5.2, type=memory+filesystem, sync=full, checksum=on, max_chunks_up=128
[2025/05/24 09:55:01] [ info] [storage] backlog input plugin: storage_backlog.1
[2025/05/24 09:55:01] [ info] [simd    ] disabled
[2025/05/24 09:55:01] [ info] [cmetrics] version=0.9.9
[2025/05/24 09:55:01] [ info] [ctraces ] version=0.6.2
[2025/05/24 09:55:01] [ info] [input:tail:tail.0] initializing
[2025/05/24 09:55:01] [ info] [input:tail:tail.0] storage_strategy='filesystem' (memory + filesystem)
[2025/05/24 09:55:01] [debug] [tail:tail.0] created event channels: read=848 write=852
[2025/05/24 09:55:01] [ info] [input:tail:tail.0] multiline core started
[2025/05/24 09:55:01] [debug] [input:tail:tail.0] flb_tail_fs_stat_init() initializing stat tail input

[2025/05/24 09:55:01] [debug] [input:tail:tail.0] inode=39406496740684990 with offset=164796 appended as D:\Work\Eclipse\Deploy\Server\logs\trade.2025-05-22.0.log
[2025/05/24 09:55:01] [debug] [input:tail:tail.0] inode=29836347532963967 with offset=64429 appended as D:\Work\Eclipse\Deploy\Server\logs\trade.2025-05-23.0.log
... (append .log files)

[2025/05/24 09:55:01] [debug] [input:tail:tail.0] 300 new files found on path '..\logs\*.log'
[2025/05/24 09:55:01] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2025/05/24 09:55:01] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2025/05/24 09:55:01] [debug] [storage_backlog:storage_backlog.1] created event channels: read=2088 write=2092
[2025/05/24 09:55:01] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 95.4M
[2025/05/24 09:55:01] [debug] [filter:modify:modify.2] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [filter:modify:modify.6] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [filter:modify:modify.7] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [filter:modify:modify.8] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [opensearch:opensearch.0] created event channels: read=2108 write=2112
[2025/05/24 09:55:01] [debug] [tls] successfully loaded certificates from windows system store.
[2025/05/24 09:55:01] [debug] [output:opensearch:opensearch.0] host=our_amazon_host port=443 uri=/_bulk index=fluent-bit type=_doc
[2025/05/24 09:55:01] [debug] [router] match rule tail.0:opensearch.0
[2025/05/24 09:55:21] [debug] [router] match rule storage_backlog.1:opensearch.0
[2025/05/24 09:55:21] [ info] [sp] stream processor started
Copy code
// fluent-bit.conf

[SERVICE]
    Flush 1
    parsers_file parsers.conf
    Daemon Off
    Log_Level debug
    storage.path .\storage
    storage.max_chunks_up 128
    storage.sync full
    storage.checksum On
    storage.delete_irrecoverable_chunks On

[INPUT]
    Name tail
    Path ..\logs\*.log
    Exclude_Path ..\logs\spring-client.log,..\logs\SERVICE_NAME_UNDEFINED.log
    Path_Key filename
    multiline.parser multiline_parser
    storage.type filesystem
    storage.pause_on_chunks_overlimit Off

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser logfmt_parser
    Reserve_Data On
    Preserve_Key True

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser message_parser
    Reserve_Data On

[FILTER]
    Name modify
    Match *
    Add host ${COMPUTERNAME}

[FILTER]
    Name parser
    Match *
    Key_Name filename
    Parser service_name_parser
    Reserve_Data On
    Preserve_Key True

[FILTER]
    Name parser
    Match *
    Key_Name mdc_fields
    Parser json_parser
    Reserve_Data On
    Preserve_Key True

[FILTER]
    Name lua
    Match *
    script dev_prefix.lua
    call add_dev_service

[FILTER]
    Name modify
    Match *
    Rename log message

[FILTER]
    Name modify
    Match *
    Remove filename

[FILTER]
    Name modify
    Match *
    Remove mdc_fields

[OUTPUT]
    Name opensearch
    Match *
    Host <our_amazon_host>
    Port 443
    tls On
    Logstash_Format On
    Logstash_Prefix_Key dev-service
    Replace_Dots On
    HTTP_User ${AWS_OPENSEARCH_USER}
    HTTP_Passwd ${AWS_OPENSEARCH_PASSWORD}
    Type _doc
    Suppress_Type_Name On
    Retry_Limit 5
    storage.total_limit_size 1G
    Trace_Error On
Copy code
// parsers.conf

[MULTILINE_PARSER]
    name multiline_parser
    type regex
    flush_timeout 2000
    rule "start_state" "^time=" "cont"
    rule "cont" "^(?!time=).*" "cont"

[PARSER]
    Name logfmt_parser
    Format regex
    Regex time=(?<time>[^ ]+ [^ ]+) thread=(?<thread>.*?) level=(?<level>[A-Z]+) logger=(?<logger>.*?) line=(?<line>\d+)
    Time_Key time
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Offset +0900

[PARSER]
    Name message_parser
    Format regex
    Regex message=(?<message>[\s\S]*?)(?=(?: mdc_fields=|\z))(?: mdc_fields=(?<mdc_fields>\{.*?\}))?

[PARSER]
    Name json_parser
    Format json

[PARSER]
    Name service_name_parser
    Format regex
    Regex ^.*[\\/](?<service>[^\\/\\.]+)\.\d{4}-\d{2}-\d{2}[-.]\d+\.log$
Copy code
// dev_prefix.lua

function add_dev_service(tag, timestamp, record)
    local svc = record["service"]
    if svc ~= nil then
        record["dev-service"] = "dev-" .. svc
    end
    return 1, timestamp, record
end
Copy code
// folder structure 

fluent-bit
- dev_prefux.lua
- fluent-bit.conf
- fluent-bit.exe
- parsers.conf
- storage
  - tail.0
    - 17808-1746146621.209801500.flb
Everything works well in general.
But when I put corrupted chunk file in tail.0 directory and restart the fluent-bit with
fluent-bit.exe -c fluent-bit.conf
, fluent-bit is paniced without error log.