One basic question about data preparation (for ingestion into Pinot) - how do you combine e.g. multiple Kafka topics into one table in Pinot so that you can query them as one - without having JOINs? Is there any way to do it without heavy upfront stream processing using e.g. Kafka Streams/ksqlDB/Flink/Materialize etc.?

Do you want to simply union thebtwo streams or perform some kind of join across the two streams

A union would be a good start (basically putting a bunch of Kafka topics into one table in Pinot), of course some kind of join would be even better. I'd just like to avoid having to use stream processing for this and just pull the data from various Kafka topics into Pinot and go from there 🙂

It’s a samza job that does join and writes back to kafka

Thanks! And what if I don't want to add a stream processing component to my architecture - what options would you recommend?

Depends on is it a join or simple union of two topics

So in our case we have e.g. one topic of daily aggregated Twitter sentiments and one topic of daily aggregated copper prices (simplified example). It could be that one of the time series has a different starting point compared to the other - e.g. the Twitter sentiments would start in 2015 and the copper prices in 1990. Would it be possible to bring the data starting from 2015 together into one Pinot table with the union operation?

You can write a plug-in that is a composite consumer across multiple topics

Cool thanks - I'll try that 😀

happy to help if you can share the PR or a github.

We're not yet there - I first have to drag parts of our team into the Kafka rabbit hole to get the tweets and reddit posts etc. on Kafka, then we have to bring it together with the commodity price data...