Hi team, I have configured Fluent Bit to read even...
# fluent-bit
d
Hi team, I have configured Fluent Bit to read events from a directory where files are rotated based on size or time (whichever occurs first). Fluent Bit tails all these files to capture incoming events. We also use the hot-reload feature to apply new configurations daily at 12:01 AM. While the reload occasionally succeeds, it often hangs during the process. Could you please help me identify what I might be doing incorrectly? Every midnight we update config to read from different directory as below: Today: /mnt/data/2025/08/19/events/*.njson Tomorrow it will be as : /mnt/data/2025/08/20/events/*.njson Please find below FluentBit's configuration.
[SERVICE] Flush 3 Grace 5 Daemon Off Log_Level Info HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 Health_Check On Parsers_File /mnt/fluentbit/conf.d/parsers.conf Storage.metrics On Storage.sync Normal Storage.checksum Off Storage.backlog.mem_limit 2048M Storage.path /data/fluentbit/storage Storage.max_chunks_up 512 Storage.backlog.chunk_size 2M Storage.delete_oldest On Workers 24 Hot_Reload On [INPUT] Name tail Path /mnt/data/2025/08/19/events/*.njson Path_Key filepath Tag daily.logs DB /data/fluentbit/db/fluentbit.db DB.Sync Normal DB.compare_filename true Parser json Mem_Buf_Limit 1024M Buffer_Chunk_Size 8M Buffer_Max_Size 32M Skip_Long_Lines Off Skip_Empty_Lines On Refresh_Interval 1 Ignore_Older 1h Rotate_Wait 60 Read_from_Head On Inotify_Watcher On Exit_On_Eof Off Multiline Off Storage.type filesystem threaded true [FILTER] Name modify Match daily.logs Rename log message [FILTER] Name lua Match daily.logs script /mnt/fluentbit/conf.d/transform.lua call process [OUTPUT] Name http Match daily.logs Host clickhouse-chproxy.clickhouse-cluster.svc.cluster.local Port 8080 URI /?query=INSERT%20INTO%20test_events.events_distributed%20FORMAT%20JSONEachRow&async_insert=1&wait_for_async_insert=0 Format json json_date_key timestamp json_date_format iso8601 Header Content-Type application/json Workers 24 Retry_Limit 5 net.connect_timeout 20 net.keepalive On net.keepalive_idle_timeout 300 net.keepalive_max_recycle 2000 net.max_worker_connections 0 storage.total_limit_size 32GB Compress gzip http_user $FLUENTBIT_USERNAME http_passwd $FLUENTBIT_PASSWORD EOF
image.png
HI @eduardo I would appreciate it if you could suggest how we should proceed
l
hi Deepak. What's exactly the issue you want to address? I assume it's the reload... how are you doing the reload? when you say it hangs, what exactly happens? what's the log for that? which versio are you using?
d
Hi @lecaros thank you for looking my issue please find the details as below: When the reload API (
curl -s <http://127.0.0.1:2020/api/v1/health>
) is invoked, the context does not restart. Based on the log messages, it appears to hang after flushing six times on the HTTP output. Log writing also stops at this point, so the logs below are the last entries we receive.
Copy code
[2025/08/19 00:00:11] [engine] caught signal (SIGHUP)
[2025/08/19 00:00:12] [ info] reloading instance pid=1 tid=0x7fef45c56ec0
[2025/08/19 00:00:12] [ info] [reload] stop everything of the old context
[2025/08/19 00:00:12] [ warn] [engine] service will shutdown when all remaining tasks are flushed
[2025/08/19 00:00:12] [ info] [input] pausing storage_backlog.1
[2025/08/19 00:00:12] [ info] [engine] service has stopped (0 pending tasks)
[2025/08/19 00:00:12] [ info] [input] pausing storage_backlog.1
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #0 stopping...
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #0 stopped
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #1 stopping...
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #1 stopped
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #2 stopping...
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #2 stopped
[2025/08/19 00:00:12] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=18005413420718882816 watch_fd=1
[2025/08/19 00:00:12] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=13034560727803756544 watch_fd=2
[2025/08/19 00:00:12] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=16505741133083574272 watch_fd=3
[2025/08/19 00:00:12] [ warn] [engine] service will shutdown when all remaining tasks are flushed [2025/08/16 00:00:12] [info] [input] pausing storage_backlog.1
[2025/08/19 00:00:12] [info] [task] tail/tail.0 has pending task(s):
[2025/08/19 00:00:12] [info] [task] storage_backlog/storage_backlog.1 has pending task(s): [2025/08/16 00:00:12] [info] [engine] pending chunk count: memory=0, filesystem=2; grace_timer=1
[2025/08/19 00:00:12] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200 
[2025/08/19 00:00:12] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200 
[2025/08/19 00:00:12] [ info] [input] pausing storage_backlog.1
[2025/08/19 00:00:13] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200 
[2025/08/19 00:00:13] [info] [engine] pending chunk count: memory=0, filesystem-2; grace_timer-2
[2025/08/19 00:00:13] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200 
[2025/08/19 00:00:13] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200 
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status-200 
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200 
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200 
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200

---- after this there was no log and also no data ingestion was happening---
Some days reloads happens properly.
Version: Fluent Bit v4.0.6
e
@Deepak we got some recent fixes in reload/shutdown phases, pls try to repro with v4.0.8
👍 1
d
Sure @eduardo let me try this version.
Hi @eduardo, We deployed version 4.0.8 and are now seeing the following logs. It appears that the engine is continuously waiting to flush chunks from the filesystem. Is there a setting to define a maximum limit for
grace_timer
, so that on restart it sends the remaining chunks?
Copy code
[2025/08/26 09:55:19] [ info] [engine] pending chunk count: memory=0, filesystem=24; grace_timer=9970
[2025/08/26 09:55:20] [ info] [engine] pending chunk count: memory=0, filesystem=24; grace_timer=9971
[2025/08/26 09:55:21] [ info] [engine] pending chunk count: memory=0, filesystem=24; grace_timer=9972
Scenario when this happened: 1. FB was sending data over HTTP to clickhouse 2. did shutdown of the Clikchouse server 3. called hot reload using HTTP api 4. then FB started writing the above log endless 5. Further hotreload doesn't received Below new config add: [Service]
Copy code
Hot_Reload.Ensure_Thread_Safety true
Shutdown_Grace  5             Storage.delete_irrecoverable_chunks    on
Storage.backlog.flush_on_shutdown Off
Copy code
[OUTPUT]
Retry_Limit no_limits
e
hi @Deepak, right now we are working in a fix for the internal http client that can be potentially related
d
Thank you so much @eduardo will retry once this go live.