This message was deleted.

To have this be reliable, I would just create a consumer worker that reads from rabbitmq and push to either Kafka or Kinesis. You can use Flink for this as well.

Tranquility is no longer being worked on, it has been replaced by kafka and kinesis ingestions which provide lossless exactly once guarantees.

We are currently using kafka, but are looking to reduce the cpu/memory footprint and operational complexity. If it's going through rabbitmq first the exactly once guarantees don't exactly accomplish anything anyways.

If you did patch Tranquility to work with newer version of Druid. please open source that work, my team could definitely use it as well 😉

Have you looked at Redpanda? It is Kafka-compatible and claims to differentiate from Kafka in the ways you mentioned (hw footprint, operations)

Redpanda does seem interesting. I'm already partway through making a rabbitmq-stream ingestor. If that doesn't work out, it seems like this would be worth a shot.