https://pinot.apache.org/ logo
#general
Title
# general
s

Sahar

01/20/2022, 2:39 PM
Hi 👋 is there a way to provide more than one kafka topic for ingestion in a table definition ("stream.kafka.topic.name") ? I provided a comma separated list but am getting a 500 error saying my kafka topic is not valid. I have two identical topics each from a different db shard that I would like them both landing in the same table
m

Mayank

01/20/2022, 3:31 PM
Hi, currently a table can consuming from only one topic. However same topic can be consumed by multiple tables.
s

Sahar

01/20/2022, 3:35 PM
ok thanks, will find a workaround
s

Sajjad Moradi

01/20/2022, 5:39 PM
There's a design doc to support multi-topic consumption for one table. I'll soon share it here, but as of now as Mayank mention Pinot only can consume from one topic.
s

Sahar

01/20/2022, 5:40 PM
awesome, thanks. Do you know what the timelines look like for its release?
s

Sajjad Moradi

01/20/2022, 5:42 PM
I can't say for sure. First the design should be reviewed and then implementation follows
👍 1
s

Subbu Subramaniam

01/20/2022, 5:48 PM
@User so that we can address the right use case, I have a couple of questions. I am guessing that your pinot schema is a subset of each of your topic schemas. Am I right? There is currently no plan to support multiple schemas in pinot.
s

Sahar

01/20/2022, 5:52 PM
@User no, the two topics have identical structure/schemas. The source tables for both topics are the exact same, they just reside on two different shards/db servers, so coming through debezium I will have two topics with different names and same schema (one per shard)
reading more on debezium documentation, looks like there is a way to make two source tables CDC events to be streamed into the same topic so I can configure it at debezium level
s

Subbu Subramaniam

01/20/2022, 6:01 PM
Pardon my ignorance, but what is debezium? A link will be great. And yes, at present you will need to combine the two topics externally before feeding them into pinot (as a third topic, I guess). I think this is what you are planning to do.
s

Sahar

01/20/2022, 6:05 PM
debezium is the source kafka connector for databases such as Mysql (the producer), it consumes the Mysql binlogs and produces Jason/Avro messages on kafka topics: I'm referring to this document https://debezium.io/documentation/reference/stable/transformations/topic-routing.html
m

Mayank

01/20/2022, 7:13 PM
@User Yes, configuring debezium to pump the events into same topic would be the clean approach here.
s

Sahar

01/20/2022, 7:23 PM
thanks all for the help
👍 1