@kapa.ai is it possible to get the partition and offset for the Kafka topic into the ingested data from a Kafka source connector?

The knowledge sources do not explicitly mention a built-in configuration option to include the Kafka partition and offset metadata in the ingested records from the Kafka source connector. Looking at the source code for both

JsonFormat

and

AvroFormat

, the records emitted by the connector only include: •

stream

(the topic name) •

emittedAt

(the timestamp) •

data

(the message value, wrapped in a

value

field)

return new AirbyteMessage()
    .withType(AirbyteMessage.Type.RECORD)
    .withRecord(new AirbyteRecordMessage()
        .withStream(record.topic())
        .withEmittedAt(Instant.now().toEpochMilli())
        .withData(Jsons.jsonNode(ImmutableMap.builder().put("value", record.value()).build())));

The

ConsumerRecord

object does contain

.partition()

and

.offset()

methods, but neither format currently maps those fields into the outgoing

AirbyteRecordMessage

. [JsonFormat source] [AvroFormat source] The official documentation for the Kafka source connector also does not mention partition or offset metadata being included in the ingested data. [Kafka source docs] In summary, this capability does not appear to be supported out of the box. Your options would be: 1. Fork/customize the connector — modify

JsonFormat.java

and/or

AvroFormat.java

to include

record.partition()

and

record.offset()

in the

ImmutableMap

passed to

.withData(...)

. 2. Open a feature request on the Airbyte GitHub to request this as a built-in option. 3. Check the Airbyte Community Slack (

slack.airbyte.com

) for any workarounds others may have implemented. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.