Hi, I am trying to get Kafka topics and their schemas from schema registry and populate Pinot in some sort of automated way. Do the schemas first need to be converted manually to this format ( https://github.com/mneedham/mastodon-stream/blob/main/pinot/schema.json )? If yes then what is the use of

“stream.kafka.decoder.prop.schema.registry.rest.url”: “<http://schema-registry:8081>”

param in table config (example taken from https://github.com/mneedham/mastodon-stream/blob/main/pinot/table.json ) ?

yeh you need to manually create the Pinot schema. You can use the

AvroSchemaToPinotSchema

command to create that for you if you don’t fancy typing it out. The schema registry URL that you referenced is used by the Kafka consumer when it’s working out how to deserialise a message - it gets a bunch of bytes and the schema helps it work out what to do with that.

thanks for

AvroSchemaToPinotSchema

, I shall try it out