Hi Team, we looking for details on whether there is a way to track record/row level metadata in Pinot - say, when the record was ingested or when the original kafka message for the record was created. For the second part we are trying to write a custom avro decoder that we can use to capture this data. But not sure if we can access the message level metadata from kafka as we only have access to the payload in the decoder.

header should be supported. i’m not sure if it’s mentioned in the docs, but it seems to be https://github.com/apache/pinot/blob/f79b618141e07c140663791959d96956ad91d811/pino[…]ain/java/org/apache/pinot/spi/stream/StreamDataDecoderImpl.java. if the field starts with

__metadata$

or

__header$

Thank you for the details. Found this documentation here: https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#extract-record-headers-as-pinot-table-columns