Yeshwanth
06/24/2025, 9:13 AM"streamIngestionConfig": {
"streamConfigMaps": [
{
"streamType": "kafka",
"stream.kafka.topic.name": "flattened_spans2",
"stream.kafka.broker.list": "kafka:9092",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "30m",
"realtime.segment.flush.threshold.segment.size": "300M"
},
{
"streamType": "kafka",
"stream.kafka.topic.name": "flattened_spans3",
"stream.kafka.broker.list": "kafka.pinot-0-nfr-setup.svc.cluster.local:9092",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "30m",
"realtime.segment.flush.threshold.segment.size": "300M"
}
]
}
But i am running into this issue.
First kafka
2025/06/20 13:21:17.528 INFO [KafkaConsumer] [otel_spans__1__0__20250620T1321Z] [Consumer clientId=otel_spans_REALTIME-flattened_spans2-1, groupId=null] Seeking to offset 0 for partition flattened_spans2-1
Second kafka
025/06/20 13:22:08.659 INFO [KafkaConsumer] [otel_spans__10001__0__20250620T1321Z] [Consumer clientId=otel_spans_REALTIME-flattened_spans3-1, groupId=null] Seeking to offset 0 for partition flattened_spans3-10001
2025/06/20 13:22:08.659 INFO [KafkaConsumer] [otel_spans__10000__0__20250620T1321Z] [Consumer clientId=otel_spans_REALTIME-flattened_spans3-0, groupId=null] Seeking to offset 0 for partition flattened_spans3-10000
the flattened_spans3 has only partitions 1-3 but the pinot server is seeking out partition number 10000 for some reason.
Can someone please guide me on where i'm going wrong with my config ?Yeshwanth
07/08/2025, 4:14 AM