@here - I am evaluating Pinot for my organization....
# generalb
Bharath Sankara
02/02/2021, 6:36 PM@here - I am evaluating Pinot for my organization.. We will be ingesting Real-time data from Kafka. What schema formats does Pinot support to ingest from Kafka? We have protobuf serialized data, is there a support for that with pinot? Thanks in advance for your help! (I am sure I will have more questions)
w
Will Briggs
02/02/2021, 6:38 PMI have only used JSON and Avro - it looks like there is an open issue for supporting protobuf: https://github.com/apache/incubator-pinot/issues/5954
Will Briggs
02/02/2021, 6:39 PMHowever, it does look like there was Protobuf support added in 0.6.0: https://github.com/apache/incubator-pinot/pull/5293
n
Neha Pawar
02/02/2021, 6:39 PM
this protobuf support is only for Offline currently
Neha Pawar
02/02/2021, 6:40 PM
from kafka we can only do json/avro
b
Bharath Sankara
02/02/2021, 6:40 PMThanks @Will Briggs @Neha Pawar
Bharath Sankara
02/02/2021, 6:40 PMis there a plan to support protos for Pinot realtime ingestion from Kafka @Neha Pawar?
n
Neha Pawar
02/02/2021, 6:41 PM
yes, we plan to
Neha Pawar
02/02/2021, 6:41 PM
how soon do you need it?
b
Bharath Sankara
02/02/2021, 6:42 PMany timeline cadence? Just need to factor that into our design... Thanks again
n
Neha Pawar
02/02/2021, 6:42 PM
@Kishore G ^
b
Bharath Sankara
02/02/2021, 6:42 PMwe are actively developing - so sooner I would say..
n
Neha Pawar
02/02/2021, 6:43 PM
also @Kartik Khare ^ who would likely be working on it
b
Bharath Sankara
02/02/2021, 9:02 PM@Kishore G @Kartik Khare - any timeline for protobuf support for pinot?