Hello, we have a setup with 2 sinks on Fluent Bit ...
# fluent-bit
n
Hello, we have a setup with 2 sinks on Fluent Bit and found out that when one of the sink has a failure, the other one stops as well, basically starting to buffer for both sinks at that point and until the failed one comes back up. We would like to pause and buffer just for the failed sink and continue streaming data on the others, so our data pipeline doesn't stop. Is this possible with Fluent Bit?
p
What's your full config?
You may need to tweak the retry configuration for the outputs so it gives up: https://docs.fluentbit.io/manual/administration/scheduling-and-retries This will mean it should discard those chunks after the retries are exhausted so they are no longer stopping new chunks once the max is reached.
And, as always, make sure you're on the latest OSS version (4.0.3 currently) šŸ™‚
If you're using filesystem buffering then there's also some other tweaks per output: https://docs.fluentbit.io/manual/administration/buffering-and-storage#output-section-configuration The whole buffering page is worth reading anyway
n
we were aiming for a solution that does not involve data loss even for the failed sinks while continuing streaming on others
p
There are always trade offs: if your output is down for too long then it will have data loss if your buffering (wherever the data is being put whilst it is offline) cannot support it. I'm not trying to argue šŸ™‚ , just saying that fundamentally no data loss but failing outputs can be a hard requirement to satisfy (e.g. your output is down for 1 day/month/year/whatever and your files rotate in 1/10/whatever minutes then you need to be able to support the data has to be buffered or lost). You can increase the number of chunks available and again the buffering page goes into the details much better than me. An alternative is to stream to another HA destination (e.g. S3) and then replay data afterwards as well.
Did you check the buffering and storage page? If not I'd really say have a look at it to get an idea of how it works and that might help you figure out the best approach.
Another alternative is to use a forward-aggregator model so just forward to another deployment that can do the buffering for each output and be controlled individually that way. It's a fairly common pattern to also offload any additional processing and scale the pipelines that way. 1. Simple forwarder picks up logs and does any basic processing before forwarding to aggregator(s) 2. Aggregator can do more processing and scale independently for each output
n
yeah, we did take a look at the docs on this. To give a bit of context, we have a setup where we have a thin fluent bit collector on app server instances that picks up the logs from files generated by a java app. This does not do any special buffering. Then we have a fluent bit aggregator where the collector will send the logs. It's on this aggregator component that we're doing the buffering. We configured it to do some memory buffering and then overflow to disk up to a certain capacity. After this capacity is reached we're ok with discarding data. This capacity will basically determine the length of the outage without data loss - let's assume we have capacity to absorb 10min of data. The expectation was that during an outage in one of the sinks, for those 10 minutes the other sinks would continue to operate normally. When the failed sink comes back up it would ingest buffered data from the outage period and eventually catch up to be on par with the other sinks.
p
Hmm yeah I think that should work, what version are you on?
The filesystem buffering should fill up whilst you are down for the inputs but outputs should continue. Once the max is reached though that means the inputs cannot ingest. You can control the individual outputs as well to limit the max storage so they do not block other outputs once it is hit.
Can you reproduce it with a simple example?
n
we're using version 4.0.3
Copy code
[SERVICE]
    daemon false
    grace 30
    http_Port 2020
    http_listen 0.0.0.0
    http_server true
    log_level info
    parsers_file /fluent-bit/etc/parsers.conf
    storage.backlog.flush_on_shutdown on
    storage.metrics true
    storage.path ${STORAGE_PATH}


[INPUT]
    name          opentelemetry
    alias         otlp-app
    listen        0.0.0.0
    port          4318
    tag_from_uri  false
    tag           app
    threaded      true
    storage.type  filesystem


[OUTPUT]
    name          opentelemetry
    alias         otlp-app-us-east-1a
    match         app*
    host          vlinsert.vlogs-us-east-1a.svc.cluster.local
    port          9481
    logs_uri      /insert/opentelemetry/v1/logs
    header        AccountID 1
    header        VL-Stream-Fields environment,hostname,role
    workers       1
    log_level     warn
    retry_limit   false
    storage.total_limit_size 10G

[OUTPUT]
    name          opentelemetry
    alias         otlp-app-us-east-1b
    match         app*
    host          vlinsert.vlogs-us-east-1b.svc.cluster.local
    port          9481
    logs_uri      /insert/opentelemetry/v1/logs
    header        AccountID 1
    header        VL-Stream-Fields environment,hostname,role
    workers       1
    log_level     warn
    retry_limit   false
    storage.total_limit_size 10G
we're deploying it via helm chart. This is the relevant config
p
There's probably something I'm missing in the understanding but if you have an example that would help. @Leonardo has a lot more experience in this area as well on the specifics. The gist seems to be: • Two outputs with filesystem buffering configured (10GiB max) • One output goes down • The other output should carry on working fine but chunks start getting buffered now • Once the storage max is hit then that's where I would expect ingestion to stop (which is on the input side) I believe you should be able to configure each output to specify the maximum it should buffer and keep that below the total so the input never stops?
You may want to set
storage.max_chunks_up
as the default is 128:
Copy code
Chunks are never much larger than 2 MB,
so with the default storage.max_chunks_up value of 128, each input is limited to
roughly 256 MB of memory.
Ah no, that's in memory. I will let Leo answer lol
šŸ‘ 1
n
Hi, am working with Nelson - I did a bit of testing today without disk buffer and with
retry_limit no_retries
on outputs. This almost gives the failover behaviour we are after in that if one output goes down it's not limiting throughput to the other output, however it still lacks some resilience for transient issues and maybe can loose data with surges. I had half a thought to add buffering downstream, roughly like this but was wondering about simpler setup that maybe gives similar behaviour:
Screenshot 2025-06-26 at 16.56.48.png
p
Yeah that's along the lines of what I suggested before around separate deployments, you get some decent benefits around being able to scale independently, monitor health of specific outputs, etc but it's a bit more complexity of course.
It also opens up the options for some more complex processing in each deployment too although that's not necessarily something you are after
l
For some reason it's not the first time I hear about this symtom where 2 unrelated sinks bound by a route go down when one of them does, I can't think of any reasoning for this behavior, it sounds like an error
n
I'm not 100% sure - the metric traces I have are a bit messy but I think the disk buffer creates back pressure even when retries are disabled - the open path didn't go to zero but the throughput was restricted. When I removed the disk buffering the open path seemed fully open while the other was closed.
p
I wonder if it still happens with an output that doesn't do full otel, e.g. one doing just logs like S3 or forward
l
Even putting together a simple repro with full otel would be a step in the right direction
Then we can weed out other factors like limits