Apache Pinot

Hi everyone,
Would like to know if it's possible to configure a real-time table, consuming from 2 different Kafka clusters ?

This would a good question to ask during next month's Contributor call as one of the topics will be multi-topic ingestion

Do you mean it is a new feature to be discussed, or is this already possible ?

iiuc, multi-topic ingestion will be in version 1.4 which will be released soon 

Not sure if it supports multiple clusters though 

If not, you would have to consider mirroring 

I don't see any mention of such multi-topic ingestion feature in the 1.4.0 pre-release note.

Seems like it was delivered in 1.3.0 release <https://github.com/apache/pinot/commit/73abb21c23d01d62f26e8f2a1eb8fddcd0f75aa2>

You're right. It was in the 1.3 release <https://docs.pinot.apache.org/basics/releases/1.3.0#multi-stream-ingestion-support-13790-design-doc>

Based on the sample config in <https://github.com/apache/pinot/pull/13790>, it appears that you can have a different broker list per topic.

Then my question is: does this work consuming from 2 topics having exactly the same name, each from a different kafka cluster ?

<@UDT7GFEG6> <@U018R9NE5EC> <@UG3BZ4ALQ> Because I see you as reviewers of this multi stream ingestion support PR, would you be able to share some details?

it's possible. one thing to note is the partitionNumber won't be the same from the same topic in two different kafka cluster.
e.g.

you have kafka topic in cluster1 and cluster2. they will have different partitionConsumers.

<@U09GWPN68VC>: as <@U028M8CAN95> mentioned, multi-topic ingestion is a feature in 1.3.0 release

Hi <@U018R9NE5EC> thanks for your feedback.
I am asking this question because I caught the following WARN logs from Pinot server which make me think multiple Kafka topics with the same name is probably not 100% covered used case:
```2025/09/24 16:10:33.240 WARN [AppInfoParser] [HelixTaskExecutor-message_handle_thread_30] Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=mytable_REALTIME-mytopic-7```
Do you think there are only minor changes like the mbeans required on the server which do not prevent it to support the use case, or there may be some other more critical changes required to fully support it ?