Hi. Pinot Team! <https://docs.pinot.apache.org/bas...
# general
c
Hi. Pinot Team! https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#extract-record-headers-as-pinot-table-columns I'm testing this function. Is this feature included in the release-0.11.0 version? In my test, the record is streamed from nifi. The 'key' and 'metadata$offset' columns in the Pinot table are always null. Is there any setting to add to Pinot or Kafka? table config :
Copy code
"tableIndexConfig": {
    "loadMode": "MMAP",
    "nullHandlingEnabled": true,
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowLevel",
      "stream.kafka.topic.name": "meetupRSVPEvents",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "localhost:19092",
      "stream.kafka.consumer.prop.auto.offset.reset": "largest",
      "realtime.segment.flush.threshold.time": "12h",
      "realtime.segment.flush.threshold.size": "10K",
      "stream.kafka.metadata.populate": "true"
    }
  },
s
This got added pretty recently. I don't think that it's supported in 0.11.0 (I checked https://github.com/apache/pinot/releases/tag/release-0.11.0) @coco Can you try it out by building from the latest master branch?
c
@Seunghyun https://pinot.apache.org/download I reinstalled the cluster with version 0.11.0 and tested it. I already restarted the cluster with version 0.11.0 already in my previous tests.
s
Oh what i meant is that this feature seems to be checked in after
0.11.0
release. So, this probably will be available from
0.12.0
c
@Seunghyun Oh I got it!! Thank you.