For stream ingestion with Kafka, only JSON format ...
# generalm
Map
06/09/2021, 9:13 PMFor stream ingestion with Kafka, only JSON format is currently supported right? The input formatslisted here https://docs.pinot.apache.org/basics/data-import/pinot-input-formats are only for batch ingestion?
x
Xiang Fu
06/09/2021, 9:14 PM
json/avro
Xiang Fu
06/09/2021, 9:14 PM
you can also write your own decoder by implementing:
org.apache.pinot.spi.stream.StreamMessageDecoderk
Kishore G
06/09/2021, 9:17 PM
thrift/protobuf etc streamsource is decoupled from the data format
m
Map
06/09/2021, 9:24 PMi see. Missed the avro one
Map
06/09/2021, 9:25 PMthought we would put all kafka decoders under
src/main/java/org/apache/pinot/plugin/stream/kafkaMap
06/09/2021, 9:26 PMthrift/protobuf etc streamsource is decoupled from the data format@User would you mind elaborating on this? You mean thrift/protobuf is available for kakfa streaming ingestion? I don't see a decoder class for them under
pinot-plugins/pinot-input-formatk
Kishore G
06/10/2021, 1:13 AM
yeah, looks like thrift/protobuf is available only for batch ingestion.. I dont see any reason why we cant add that for real-time as well