kanetla
05/22/2025, 6:07 AMstorage.delete_irrecoverable_chunks On option but .flb files are not deleted automatically.
How to solve this problem?kanetla
05/22/2025, 6:07 AM// fluent-bit.conf
[SERVICE]
Flush 1
parsers_file parsers.conf
Daemon Off
Log_Level info
storage.path .\storage
storage.max_chunks_up 128
storage.sync full
storage.checksum On
storage.delete_irrecoverable_chunks Onkanetla
05/22/2025, 6:08 AM[2025/05/22 14:48:05] [error] [storage] format check failed: tail.0/51904-1746111458.203209800.flb
[2025/05/22 14:48:05] [ info] [storage] [cio file] invalid crc32 at 5200-1745827407.145630700.flb/.\storage\tail.0\5200-1745827407.145630700.flbkanetla
05/22/2025, 6:08 AM51904-1746111458.203209800.flb file is not deleted automatically.kanetla
05/22/2025, 6:09 AMPat
05/22/2025, 9:57 AMPat
05/22/2025, 10:01 AMPat
05/22/2025, 10:02 AMkanetla
05/23/2025, 1:10 AM.flb files are automatically deleted in general situation.
But if fluent-bit is terminated unexpectedly, some .flb files are corrupted.
In this case, when I start the fluent-bit again, fluent-bit is paniced.
Here is the corrupted file.
Is there anyway to skip corrupted file when I start the fluent-bit?Leonardo Almiñana
05/23/2025, 9:01 AMLeonardo Almiñana
05/23/2025, 9:02 AMLeonardo Almiñana
05/23/2025, 9:03 AMLeonardo Almiñana
05/23/2025, 9:03 AMkanetla
05/23/2025, 9:06 AMtail.0 directory.
[2025/05/23 18:03:22] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2025/05/23 18:03:22] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2025/05/23 18:03:22] [debug] [storage_backlog:storage_backlog.1] created event channels: read=1904 write=1908
[2025/05/23 18:03:22] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 95.4M
[2025/05/23 18:03:22] [debug] [filter:modify:modify.2] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [filter:modify:modify.6] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [filter:modify:modify.7] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [filter:modify:modify.8] Initialized modify filter with 0 conditions and 1 rules
[2025/05/23 18:03:22] [debug] [opensearch:opensearch.0] created event channels: read=2064 write=2068
[2025/05/23 18:03:22] [debug] [tls] successfully loaded certificates from windows system store.
[2025/05/23 18:03:22] [debug] [output:opensearch:opensearch.0] host=search-eclipse-log-plw4vsgtpgmd73abzarvfopwla.ap-northeast-2.es.amazonaws.com port=443 uri=/_bulk index=fluent-bit type=_doc
[2025/05/23 18:03:22] [debug] [router] match rule tail.0:opensearch.0
[2025/05/23 18:03:22] [debug] [router] match rule storage_backlog.1:opensearch.0
[2025/05/23 18:03:22] [ info] [sp] stream processor startedLeonardo Almiñana
05/23/2025, 9:36 AMLeonardo Almiñana
05/23/2025, 9:37 AMLeonardo Almiñana
05/23/2025, 9:37 AMLeonardo Almiñana
05/23/2025, 9:37 AMkanetla
05/23/2025, 9:41 AMLeonardo Almiñana
05/23/2025, 9:41 AMkanetla
05/24/2025, 12:48 AMOS: Window
Version: 4.0.0
Version for git commit: 712b48d7c387c87c922075aa5556afe648e36c4bkanetla
05/24/2025, 12:57 AMfluent-bit.exe -c fluent-bit.confkanetla
05/24/2025, 12:59 AM[2025/05/24 09:55:01] [ info] Configuration:
[2025/05/24 09:55:01] [ info] flush time | 1.000000 seconds
[2025/05/24 09:55:01] [ info] grace | 5 seconds
[2025/05/24 09:55:01] [ info] daemon | 0
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info] inputs:
[2025/05/24 09:55:01] [ info] tail
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info] filters:
[2025/05/24 09:55:01] [ info] parser.0
[2025/05/24 09:55:01] [ info] parser.1
[2025/05/24 09:55:01] [ info] modify.2
[2025/05/24 09:55:01] [ info] parser.3
[2025/05/24 09:55:01] [ info] parser.4
[2025/05/24 09:55:01] [ info] lua.5
[2025/05/24 09:55:01] [ info] modify.6
[2025/05/24 09:55:01] [ info] modify.7
[2025/05/24 09:55:01] [ info] modify.8
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info] outputs:
[2025/05/24 09:55:01] [ info] opensearch.0
[2025/05/24 09:55:01] [ info] ___________
[2025/05/24 09:55:01] [ info] collectors:
[2025/05/24 09:55:01] [ info] [fluent bit] version=4.0.0, commit=712b48d7c3, pid=31780
[2025/05/24 09:55:01] [debug] [engine] coroutine stack size: 98302 bytes (96.0K)
[2025/05/24 09:55:01] [ info] [storage] ver=1.5.2, type=memory+filesystem, sync=full, checksum=on, max_chunks_up=128
[2025/05/24 09:55:01] [ info] [storage] backlog input plugin: storage_backlog.1
[2025/05/24 09:55:01] [ info] [simd ] disabled
[2025/05/24 09:55:01] [ info] [cmetrics] version=0.9.9
[2025/05/24 09:55:01] [ info] [ctraces ] version=0.6.2
[2025/05/24 09:55:01] [ info] [input:tail:tail.0] initializing
[2025/05/24 09:55:01] [ info] [input:tail:tail.0] storage_strategy='filesystem' (memory + filesystem)
[2025/05/24 09:55:01] [debug] [tail:tail.0] created event channels: read=848 write=852
[2025/05/24 09:55:01] [ info] [input:tail:tail.0] multiline core started
[2025/05/24 09:55:01] [debug] [input:tail:tail.0] flb_tail_fs_stat_init() initializing stat tail input
[2025/05/24 09:55:01] [debug] [input:tail:tail.0] inode=39406496740684990 with offset=164796 appended as D:\Work\Eclipse\Deploy\Server\logs\trade.2025-05-22.0.log
[2025/05/24 09:55:01] [debug] [input:tail:tail.0] inode=29836347532963967 with offset=64429 appended as D:\Work\Eclipse\Deploy\Server\logs\trade.2025-05-23.0.log
... (append .log files)
[2025/05/24 09:55:01] [debug] [input:tail:tail.0] 300 new files found on path '..\logs\*.log'
[2025/05/24 09:55:01] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2025/05/24 09:55:01] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2025/05/24 09:55:01] [debug] [storage_backlog:storage_backlog.1] created event channels: read=2088 write=2092
[2025/05/24 09:55:01] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 95.4M
[2025/05/24 09:55:01] [debug] [filter:modify:modify.2] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [filter:modify:modify.6] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [filter:modify:modify.7] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [filter:modify:modify.8] Initialized modify filter with 0 conditions and 1 rules
[2025/05/24 09:55:01] [debug] [opensearch:opensearch.0] created event channels: read=2108 write=2112
[2025/05/24 09:55:01] [debug] [tls] successfully loaded certificates from windows system store.
[2025/05/24 09:55:01] [debug] [output:opensearch:opensearch.0] host=our_amazon_host port=443 uri=/_bulk index=fluent-bit type=_doc
[2025/05/24 09:55:01] [debug] [router] match rule tail.0:opensearch.0
[2025/05/24 09:55:21] [debug] [router] match rule storage_backlog.1:opensearch.0
[2025/05/24 09:55:21] [ info] [sp] stream processor startedkanetla
05/24/2025, 1:01 AM// fluent-bit.conf
[SERVICE]
Flush 1
parsers_file parsers.conf
Daemon Off
Log_Level debug
storage.path .\storage
storage.max_chunks_up 128
storage.sync full
storage.checksum On
storage.delete_irrecoverable_chunks On
[INPUT]
Name tail
Path ..\logs\*.log
Exclude_Path ..\logs\spring-client.log,..\logs\SERVICE_NAME_UNDEFINED.log
Path_Key filename
multiline.parser multiline_parser
storage.type filesystem
storage.pause_on_chunks_overlimit Off
[FILTER]
Name parser
Match *
Key_Name log
Parser logfmt_parser
Reserve_Data On
Preserve_Key True
[FILTER]
Name parser
Match *
Key_Name log
Parser message_parser
Reserve_Data On
[FILTER]
Name modify
Match *
Add host ${COMPUTERNAME}
[FILTER]
Name parser
Match *
Key_Name filename
Parser service_name_parser
Reserve_Data On
Preserve_Key True
[FILTER]
Name parser
Match *
Key_Name mdc_fields
Parser json_parser
Reserve_Data On
Preserve_Key True
[FILTER]
Name lua
Match *
script dev_prefix.lua
call add_dev_service
[FILTER]
Name modify
Match *
Rename log message
[FILTER]
Name modify
Match *
Remove filename
[FILTER]
Name modify
Match *
Remove mdc_fields
[OUTPUT]
Name opensearch
Match *
Host <our_amazon_host>
Port 443
tls On
Logstash_Format On
Logstash_Prefix_Key dev-service
Replace_Dots On
HTTP_User ${AWS_OPENSEARCH_USER}
HTTP_Passwd ${AWS_OPENSEARCH_PASSWORD}
Type _doc
Suppress_Type_Name On
Retry_Limit 5
storage.total_limit_size 1G
Trace_Error Onkanetla
05/24/2025, 1:02 AM// parsers.conf
[MULTILINE_PARSER]
name multiline_parser
type regex
flush_timeout 2000
rule "start_state" "^time=" "cont"
rule "cont" "^(?!time=).*" "cont"
[PARSER]
Name logfmt_parser
Format regex
Regex time=(?<time>[^ ]+ [^ ]+) thread=(?<thread>.*?) level=(?<level>[A-Z]+) logger=(?<logger>.*?) line=(?<line>\d+)
Time_Key time
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Offset +0900
[PARSER]
Name message_parser
Format regex
Regex message=(?<message>[\s\S]*?)(?=(?: mdc_fields=|\z))(?: mdc_fields=(?<mdc_fields>\{.*?\}))?
[PARSER]
Name json_parser
Format json
[PARSER]
Name service_name_parser
Format regex
Regex ^.*[\\/](?<service>[^\\/\\.]+)\.\d{4}-\d{2}-\d{2}[-.]\d+\.log$kanetla
05/24/2025, 1:02 AM// dev_prefix.lua
function add_dev_service(tag, timestamp, record)
local svc = record["service"]
if svc ~= nil then
record["dev-service"] = "dev-" .. svc
end
return 1, timestamp, record
endkanetla
05/24/2025, 1:05 AM// folder structure
fluent-bit
- dev_prefux.lua
- fluent-bit.conf
- fluent-bit.exe
- parsers.conf
- storage
- tail.0
- 17808-1746146621.209801500.flbkanetla
05/24/2025, 1:06 AMkanetla
05/24/2025, 1:06 AMfluent-bit.exe -c fluent-bit.conf, fluent-bit is paniced without error log.