How can we design and implement a dual data center (DC–DR) setup for Kafka output in Fluent Bit, ensuring that the output dynamically switches to the DR Kafka cluster only when the primary DC Kafka system becomes unreachable, and automatically reverts back once the DC is available again?

probably you can make it combination of lua and rewrite_tag I don't think it's possible with kafka output directly. 1. with lua ping kafka primary 2. if fails add fieled with lua 3. with rewrite_tag check this filed and add new tag 4. configure to output of kafka. primary and DR and configure them based on tag accordingly.

or add a grep processor to the output instead of rewrite_tag may be better

I appreciate your assistance and respect your expertise. Is it possible to achieve this solely through configuration — without relying on Lua scripting

Lua scripting is just configuration 😄 but no I don't think there is the upstream support for Kafka. @Leonardo may be able to confirm but I think upstream only works for forward and maybe http

Give me a second to catch up

Well you can do active-passive deployment of Kafka, sort yourself to route traffic with load balancer is another alternative

yeah, again that's configuration 🙂 but that would be the way I'd probably do it - not everything is best done by FB 🙂

According to the code : 1 - HA upstream as a feature has to be explicitly supported by the plugin (

out_kafka

doesn't) 2 -

out_kafka

uses

rd_kafka

directly which means it doesn't use fluent-bits

upstream

features at all

ah yeah, if rdkafka has an option for it then potentially that's a way to do it - you can configure any option rdkafka supports directly

Either that or a very low TTL local DNS server

yup, just use `rdkafka.{property}`: https://docs.fluentbit.io/manual/data-pipeline/outputs/kafka

I couldn’t find any suitable configuration within the rdkafka properties. Would it be feasible to use HashiCorp Consul as a potential workaround, or does anyone have a more effective solution to suggest?

I don't know anything about kafka or rdkafka but wouldn't

metadata.broker.list

be the option you're looking for?