Can someone explain how Apache Pinot integrates with Confluent Schema Registry during Kafka stream ingestion? Specifically: 1. Does Pinot use Schema Registry only for deserialization of Avro/Protobuf messages, or can it automatically generate Pinot table schemas from the registered schemas? 2. If auto-generation is supported, what are the limitations or required configurations? 3. How does Pinot handle schema evolution in Schema Registry (e.g., backward/forward compatibility) during ingestion? 4. Are there any best practices for defining Pinot schemas when using Schema Registry to avoid data type mismatches? Context: I’m setting up real-time ingestion from Kafka topics with Avro schemas stored in Schema Registry and want to minimize manual schema mapping work.

Schema registry is used to decode the message. But you need to specify the Pinot side schema explicitly on what columns / transforms you may need. For schema evolution you need to modify the schema in Pinot explicitly No explicit best practices doc, as it it the same with or without schema registry.

Why not to use the schema registry to create the schema.. since it is already part of the table config?

Pinot schema may not necessarily be 1:1 mapping, plus it needs information like which columns are dimensions, metrics, etc. So while there can be automation for doing the first pass, there's almost always human intervention needed.