Hi team, I enabled "stream.kafka.metadata.populate": "true", to get below fields. These fields i have added in schema also. __key __metadata$offset __metadata$partition __metadata$recordTimestamp But on querying table __metadata$offset __metadata$partition __metadata$recordTimestamp these are populated properly but __key is coming as blank. Since my kafka event and key are avro encoded. I used following config:

"stream.kafka.decoder.prop.format": "AVRO",
"stream.kafka.decoder.prop.schema.registry.schema.name": "schema-name",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
"stream.kafka.decoder.prop.schema.registry.rest.url": "schema-url",
"stream.kafka.decoder.prop.key.format": "AVRO",
"stream.kafka.decoder.prop.key.schema.registry.schema.name": "schema-name-key",
"stream.kafka.decoder.prop.key.schema.registry.rest.url": "schema-url",

data is also properly deserialised. Only __key is blank. My guess is that below configs i added is not able to deserialise it. Is there any other way to deserialise the key?

"stream.kafka.decoder.prop.key.format": "AVRO",
"stream.kafka.decoder.prop.key.schema.registry.schema.name": "schema-name-key",
"stream.kafka.decoder.prop.key.schema.registry.rest.url": "schema-url",

Hi @Arnav - Were you able to resolve this issue, sort of facing the same issue.

Not yet resolved