https://pinot.apache.org/ logo
#general
Title
# general
m

Map

06/09/2021, 9:13 PM
For stream ingestion with Kafka, only JSON format is currently supported right? The input formatslisted here https://docs.pinot.apache.org/basics/data-import/pinot-input-formats are only for batch ingestion?
x

Xiang Fu

06/09/2021, 9:14 PM
json/avro
you can also write your own decoder by implementing:
org.apache.pinot.spi.stream.StreamMessageDecoder
k

Kishore G

06/09/2021, 9:17 PM
thrift/protobuf etc streamsource is decoupled from the data format
m

Map

06/09/2021, 9:24 PM
i see. Missed the avro one
thought we would put all kafka decoders under
src/main/java/org/apache/pinot/plugin/stream/kafka
thrift/protobuf etc streamsource is decoupled from the data format
@User would you mind elaborating on this? You mean thrift/protobuf is available for kakfa streaming ingestion? I don't see a decoder class for them under
pinot-plugins/pinot-input-format
k

Kishore G

06/10/2021, 1:13 AM
yeah, looks like thrift/protobuf is available only for batch ingestion.. I dont see any reason why we cant add that for real-time as well