If I am not mistaken, Pinot's integration with kaf...
# generalm
Map
07/27/2021, 8:41 PMIf I am not mistaken, Pinot's integration with kafka schema registry is only for avro decoder? Can pinot read table schemas directly from Kafka schema registry?
m
Mayank
07/27/2021, 9:03 PM
Yes. For JSON, we simply use the table config
m
Map
07/27/2021, 9:10 PMHow do you manage/automate the schema changes? Do you have a process that takes the canonical schema and then translate it into a Pinot’s table schema?
k
Kishore G
07/28/2021, 3:37 AM
@User Can you elaborate on what is the format of data in Kafka
m
Map
07/28/2021, 8:58 PMthey are serialized json messages
k
Kishore G
07/28/2021, 10:08 PM
Got it! this should be easy to add the ConfluentJsonDecoder. Do you want to take a stab at it.. we can help you
m
Map
07/28/2021, 10:27 PMEven if it is possible we still need to manage Pinot table schemas separately right?
k
Kishore G
07/29/2021, 12:56 AM
yes. Are you looking at updating pinot schema automatically as the schema in schema registry gets updated?
Kishore G
07/29/2021, 12:56 AM
If yes, that seems like a pretty cool idea
m
Map
07/29/2021, 1:25 AMyeah dont know if we have considered it
k
Kishore G
07/29/2021, 2:55 AM
We haven’t.. I think one of the reasons was that Pinot did not support query nested columns
Kishore G
07/29/2021, 2:55 AM
But with json udf and json index, this might be a cool idea