Hi All, is it possible to ingest data from several Kafka clusters into one dataSource using Kafka supervisors?

topics: yes clusters: not that I have seen

yep. I saw topicPattern option but what if we have several independent Kafka clusters and need to gather it into one druid dataSource

You might try with concurrent append and replace... I haven't tried https://druid.apache.org/docs/latest/ingestion/concurrent-append-replace#use-concurrent-locks-in-ingestion-jobs

Good option if I could run 2 superviosrs for 1 dataSource. But it seems impossible. ioConfig option is also Object, not Array, so I can't pass several kafka configs into one supervisor....

I was thinking multiple supervisors

according to

<https://druid.incubator.apache.org/docs/latest/ingestion/supervisor/#supervisor-spec>

Once started, the supervisor persists in the configured metadata database. There can only be one supervisor per datasource. Submitting a second supervisor spec for the same datasource overwrites the previous one.

ah. very good point.

Hi, Hope you're doing well, did you find a solution for this? actually we're working on something similiar and want to ingest data from 3 different kafka servers into 1 data source. Your response would be really appreciated. Thank you.