Hello, This is my first time reaching out for hel...
# fluent-bit
t
Hello, This is my first time reaching out for help. I have a proxy server running Fluent Bit 4.0.4 that aggregates metrics in Prometheus format from multiple servers. These metrics are then forwarded using the Forward protocol over TLS to Fluent Bit agents (also version 4.0.4) running on EC2 instances behind an AWS Network Load Balancer (NLB). The issue I’m encountering is that Fluent Bit randomly terminates some connections from the servers sending metrics. Once a connection is dropped, it is not reestablished unless I manually restart the Fluent Bit service. I've experimented extensively with various networking parameters, but the behavior persists — some servers stop sending metrics after just a few minutes. Log forwarding to Fluentd using the Forward protocol is working without any issues. I considered forwarding the metrics to Fluentd instead, but unfortunately, my tests showed it's incompatible with Prometheus-format metrics. Do you have any suggestions? Could this be a bug in Fluent Bit? Follow some threads the may be related to this behavior: https://github.com/fluent/fluent-bit/issues/7434 https://github.com/fluent/fluent-bit/issues/6485 Thanks in advance.
p
Do you have debug logs of the fluent bit instance when it terminates?
t
Hi Pat, Thanks for your response. Today, I spent more time troubleshooting the issue with forwarding metrics using the
forward
protocol (Fluent Bit → Fluent Bit) to AWS, but unfortunately, it's still not working as expected. On the other hand, the logs are being successfully sent via Fluent Bit to Fluentd. I can confirm that all expected servers appear on the Fluentd instance in AWS. Below is the current topology and configuration setup I’m using: Fluent Bit Agents] [proxy aggregator] [AWS NLB] [EC2] ------------------------ non-TLS -----> Fluent Bit ----------- TLS -----> AWS NLB -----> Fluent Bit (Metrics) fluent-bit 1 \ fluent-bit 2 \-> Fluentd (Logs) – OK fluent-bit 3 fluent-bit 4 fluent-bit 5 fluent-bit 6 The Fluent Bit instance running in AWS (receiving metrics from the proxy server) behaves inconsistently, sometimes it receives metrics from some agents, but others randomly drop off and never reconnect unless restarted.
One additional note: I configured FluentD in AWS to receive metrics using the TCP input plugin, since FluentD does not natively support Prometheus-formatted data. Although the incoming data appeared as encoded or non-human-readable characters, I was able to confirm that all expected servers were successfully sending their metrics. Based on this, it appears the issue may be related to the
in_forward
plugin in Fluent Bit.
Hi @Pat, Do you have any updates? This is something I really need to address internally, given the large scope of the project I’m managing. Is there anything I can do to help speed up the analysis?
p
I'm not looking at it I'm afraid, I would raise an OSS issue with full details