https://linen.dev logo
Join Slack
Powered by
Hi everyone! I have a question about Fluent Bit's ...
# fluent-bit
m
Hi everyone! I have a question about Fluent Bit's retry engine. Does Fluent Bit have any internal limit on the number of retry jobs/tasks, even when 
Retry_Limit
is set to 
no_limits
? I'm trying to understand whether there is any engine-level behavior that causes an output (or input) plugin to stop sending new logs when the retry queue is full, effectively waiting until there's enough space for the next retry task. If such a mechanism exists, could you point me to the relevant documentation or source code? Thanks!
l
If nothing changed since last time I worked on it there is no such thing. Of course if you have set a specific storage limit and you run out of space fluent-bit will drop the oldest viable chunk in order to ingest new data but that's it.
Inputs have nothing to do with retries though
All inputs do is append the data they receive (or generate) to a chunk (a datafile in memory or disk) and then when the chunkis full (or the flush timer hits if you have a single chunk) it's sealed and scheduled to be delivered.
One limit you need to keep in mind is the concurrently scheduled flush limit but that doesn't mean that chunks won't be retried, it means they'll have to wait for a slot to be free to do so
Again, I could be wrong since I haven't been an active developer for a long time but I suspect that subsystem may not have changed since I left
m
Thanks @Leonardo . I found this github.com/fluent/fluent-bit/blob/…/flb_config.h#…
Copy code
#define FLB_CONFIG_DEFAULT_TASK_MAP_SIZE_LIMIT  16384
This is a task map size limit. I believe it is used as fluent-bit engine's task-ID allocation pool. Now, my understanding is that when 16 384 tasks are live (a mix of in-flight flushes and pending retries), the dispatch loop stops converting input chunks into tasks. New input records pile up and are not dispatched. Since retrying tasks hold slots until their retry succeeds, a flood of failed retries under 
no_limits
can exhaust all 16 384 slots and block new data from being flushed entirely. Is my understanding correct?
l
In a way, I mean, in a realistic scenario you won't have 16k concurrent tasks (especially for a single route) but even if you did (which would mean you have constantly failing retries) fluent-bit would only drop the oldest full chunks when trying to allocate new chunks (if you have exceeded the storage limit)
If your endpoint is down for a really long time then yeah you could get into that situation but that would be the regular chunk rollover mechanism, it wouldn't be any kind of lock or malfunction
Unless I misunderstood you
IDK if the circuit breaker implementation has been finally merged but if it has then that's your best bet for this situation
m
Well, it looks like Fluent Bit eventually stops sending data - it seems to stall somehow, at least from what I can tell. Here's what I did: I configured my output plugin with 
Retry_Limit false
and intentionally made my endpoint fail, so it continuously retries sending data, just as you observed. I also configured a dummy input to continuously generate data and let it run for a while. Fluent Bit is configured to use filesystem storage. After some time, Fluent Bit stops sending data altogether, and the dummy input plugin also stops generating new data. These are the last few log messages I see before everything appears to stall:
Copy code
[2026/07/08 22:37:04.276] [ warn] [engine] failed to flush chunk '98347-1783541861.892249000.flb', retry in 681 seconds: task_id=4429, input=dummy.1 > output=logger-agent-plugin.0 (out_id=0)

[2026/07/08 22:37:09.279] [ info] [task] re-schedule retry=0xb50186e50 9048 in the next 160 seconds
[2026/07/08 22:37:09.279] [ info] [task] re-schedule retry=0xb501369a0 8428 in the next 268 seconds
[2026/07/08 22:37:09.287] [ info] [task] re-schedule retry=0xb500d5c50 7688 in the next 320 seconds
[2026/07/08 22:37:09.287] [ info] [task] re-schedule retry=0xb500b8600 7408 in the next 296 seconds
[2026/07/08 22:37:09.287] [ info] [task] re-schedule retry=0xb500d61f0 7698 in the next 577 seconds
[2026/07/08 22:37:09.287] [ info] [task] re-schedule retry=0xb5006aa00 6868 in the next 15 seconds
[2026/07/08 22:37:09.287] [ info] [task] re-schedule retry=0xb5010a6d0 8068 in the next 160 seconds
[2026/07/08 22:37:09.287] [ info] [task] re-schedule retry=0xb501b9ce0 9388 in the next 170 seconds
It did not exceed the storage limit though:
Copy code
2026-07-08T20:37:04.255776008Z fluentbit_output_chunk_available_capacity_percent{name="logger-agent-plugin.0"} = 95.416056400000002
95% still available
Health check status is 
ok
After these log messages above, there are no more logs, no more retries, and no new records generated by the dummy input plugin.
l
That's interesting. I'd send a HUP signal to check the internal stats to see what's going on and also try setting fluent-bits log level to debug to see what's happening
Can you share your config file?
m
Yes, I can, but bear in mind that I'm using my own custom Go based plugin. Give me a sec
Copy code
[SERVICE]
  Flush                   1
  Log_Level               info
  Daemon                  off
  Parsers_File            /Users/mila/dev/fluent-bit/conf
  Plugins_File            /Users/mila/dev/logger-agent-plugin/plugins.conf
  HTTP_Server             On
  HTTP_Listen             0.0.0.0
  HTTP_Port               2020
  Hot_Reload              On
  Health_Check            On
  HC_Errors_Count         5
  HC_Retry_Failure_Count  5
  HC_Period               60
  storage.path            /Users/mila/tmp/fluent-bit-cache
  storage.max_chunks_up   250
  storage.metrics         On

[INPUT]
  Name  dummy
  Dummy {"message": "{\"msg\":\"TEST MESSAGE 00\"}"}
  Rate  10
  storage.type        filesystem


[OUTPUT]
  Name              logger-agent-plugin
  Match             *
  Retry_Limit       false

  # Connection
  Target_Host localhost
  Target_Port 8081
  Target_Path /v1/logs

  # Buffer storage
  storage.total_limit_size 500M
  storage.type filesystem
The output plugin is really simple, it establishes ws connection against my server and sends Compressed PackedForwardMessage, nothing else.
Here are the metrics that might be useful in this case:
Copy code
2026-07-08T19:14:50.362581899Z fluentbit_output_errors_total{name="logger-agent-plugin.0"} = 0
2026-07-08T20:37:04.255776008Z fluentbit_output_retries_total{name="logger-agent-plugin.0"} = 77773
2026-07-08T19:14:50.362581899Z fluentbit_output_retries_failed_total{name="logger-agent-plugin.0"} = 0
2026-07-08T19:14:50.362581899Z fluentbit_output_dropped_records_total{name="logger-agent-plugin.0"} = 0
2026-07-08T20:37:04.255776008Z fluentbit_output_retried_records_total{name="logger-agent-plugin.0"} = 576029

2026-07-08T20:37:04.275847100Z fluentbit_storage_chunks = 11897
2026-07-08T20:37:04.275847100Z fluentbit_storage_mem_chunks = 10797
2026-07-08T20:37:04.275847100Z fluentbit_storage_fs_chunks = 1100
2026-07-08T20:37:04.275847100Z fluentbit_storage_fs_chunks_up = 249
2026-07-08T20:37:04.275847100Z fluentbit_storage_fs_chunks_down = 851
If I count number of *.flb files in my storage path it shows 1100
l
Seems like you're running a single thread also so you shouldn't exceed the task limit
But I don't remember how the go proxy handles green threads
I mean, coroutines
It probably doesn't
So in order to rule out a systemic issue I'd just use the HTTP output plugin (is there a websocket plugin?)
Since your endpoint will be down that doesn't really matter
What's important is to determine if the failure is at the core or at an outer layer
It could be the go plugin proxy code or something else
Oh wait
One thing that's really peculiar is that you say that when the fault state is reached ingestion is also paused
So one thing you can do is set the output plugin to threaded mode by settings 
workers
to 
1
That way the main thread will run the main dispatcher code and the input plugin (unless you set the input plugins 
threaded
attribute to 
on
which you can but don't do it at the same time as the previous test so you can isolate the changes and make it easier to pinpoint which one changes the outcome)
m
Yeah, I could use a simple HTTP output plugin as well, since it doesn't matter really.
One thing that's really peculiar is that you say that when the fault state is reached ingestion is also paused
Yes, so no more logs, no more retries and no new records generated at least according to fluent-bit metrics (
curl loclhost:8081/api/v2/metrics
)
ok, you gave me some ideas and things to test.
l
Please priroritize changing the output thread count
That'd be much more meaningful
m
Yes, that's what I'm going to do first
l
Cool, let me know how it goes, I'm curious
m
Sure, thanks for you help. I'll perform the tests you suggested tomorrow, since it's quite late here. I appreciate your time and help 🙇‍♂️
l
Any luck?
m
Hi @Leonardo yes 🙂 . I configured the output plugin (still my custom Go-based output plugin) to run in threaded mode by setting 
workers
to 
1
. I wasn't able to reproduce the issue. I didn't observe any pipeline stalls, and I can see that the filesystem storage is gradually filling up (with less and less free space available).
l
Let me know how it goes
I wonder if there's somethng go specific that's causing it to block.
If you go back to single thread mode, once it enters the failure condition you can attach with gdb and list the threads to see what the thread statuses and code locations are
That way if go is blocking the thread you'll be able to easily identify it
11 Views
Open in Slack
PreviousNext
Powered by