Apache Pinot

Hello Team,

I am new to pinot world. Currently in my project we use Kafka as source and use spark streaming and create dataframe using readStream and write the data to hbase . 

Now would like to explore more on pinot .So am checking now whether using spark and Kafka , is it possible to achieve the same ?

What is the use case you are trying solve? Rather than trying to fit Pinot in your existing architecture, better to understand the requirements and how to best solve it using Pinot 

I have streaming data in kafka . We wanted to consume that message using spark and write data into pinot. End user should be able to query the realtime data and get the desired output asap.

Convo continued here <https://apache-pinot.slack.com/archives/C01H1S9J5BJ/p1691643158666599>

Hi <@U060LT3UEQK>, per our PM, you can write to Pinot using the example in this repo <https://github.com/apache/pinot/tree/master/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark-3.2>

Thanks <@U054FC97BLH>..I will have look into this.

I'll be trying this myself and publishing it. I'll paste here when complete.

<@U054FC97BLH> any chance that you have some time to explore on this

Hi <@U060LT3UEQK> not yet. I have it this week to do. Apologies for the delay.

Hi <@U060LT3UEQK>, after speaking with the PM, it turns out that the specific way you want to ingest data into Pinot from spark is still being worked on.  I think because the integration with streaming platforms like Kafka is simple and robust that we haven't had much request for a Pinot spark sink connector. I'll let them know that you're interested in it. Thanks