https://pinot.apache.org/ logo
#general
Title
# general
k

Kaustabh Ganguly

06/01/2021, 2:24 PM
I'm a fresh CS grad and just exploring things. I am new to streaming data, kafka and pinot. I want to merge batched data and streaming data and use pinot on top of it. My solution is to use Kafka connect as it's an ideal solution for merging batched and streaming data into topics & partitions. So my pipeline is basically using kafka for merging and then using pinot for streaming from kafka. Is there a better solution that comes across anyone's mind ? Please correct me if there's any fallacy in my logic.
m

Mayank

06/01/2021, 3:28 PM
Since Pinot can ingest data from offline directly, you could simply have Pinot ingest from a separate offline pipeline as well as Kafka stream.
👍 1
k

Kaustabh Ganguly

06/01/2021, 3:29 PM
Thanks. Trying that out. Will ask here if I have some doubts.
m

Mayank

06/01/2021, 3:29 PM
Yes, feel free to ask any questions here
Also see if you can find your answer ^^. If not, perhaps we can improve the docs
👍 1
k

Kaustabh Ganguly

06/01/2021, 3:31 PM
Sure thing.