Hi 👋 is there a way to provide more than one kafka topic for ingestion in a table definition ("stream.kafka.topic.name") ? I provided a comma separated list but am getting a 500 error saying my kafka topic is not valid. I have two identical topics each from a different db shard that I would like them both landing in the same table

Hi, currently a table can consuming from only one topic. However same topic can be consumed by multiple tables.

ok thanks, will find a workaround

There's a design doc to support multi-topic consumption for one table. I'll soon share it here, but as of now as Mayank mention Pinot only can consume from one topic.

awesome, thanks. Do you know what the timelines look like for its release?

I can't say for sure. First the design should be reviewed and then implementation follows

@User so that we can address the right use case, I have a couple of questions. I am guessing that your pinot schema is a subset of each of your topic schemas. Am I right? There is currently no plan to support multiple schemas in pinot.

@User no, the two topics have identical structure/schemas. The source tables for both topics are the exact same, they just reside on two different shards/db servers, so coming through debezium I will have two topics with different names and same schema (one per shard)

Pardon my ignorance, but what is debezium? A link will be great. And yes, at present you will need to combine the two topics externally before feeding them into pinot (as a third topic, I guess). I think this is what you are planning to do.

debezium is the source kafka connector for databases such as Mysql (the producer), it consumes the Mysql binlogs and produces Jason/Avro messages on kafka topics: I'm referring to this document https://debezium.io/documentation/reference/stable/transformations/topic-routing.html

@User Yes, configuring debezium to pump the events into same topic would be the clean approach here.

thanks all for the help