Shounak Kulkarni
06/29/2020, 4:33 PMMayank
Shounak Kulkarni
06/29/2020, 4:35 PMPradeep
06/30/2020, 12:48 AMstream.kafka.zk.broker.url
missing, added it later on.
I have number of replicas set to 1
and no tagsOverrideConfig
"streamType": "kafka",
"stream.kafka.consumer.type": "LowLevel",
"stream.kafka.topic.name": "<TOPIC>",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "<node1>:9092,<node2>:9092",
"stream.kafka.zk.broker.url": "<zk>:2191",
Pradeep
06/30/2020, 1:07 AM"numSegmentsQueried": 14,
where there are 64 kafka partitions and also /segments/{tableName} shows ~68 segmentsPradeep
06/30/2020, 1:09 AMJackie
06/30/2020, 1:41 AMKishore G
Pradeep
06/30/2020, 1:52 AMJackie
06/30/2020, 2:06 AMPradeep
06/30/2020, 2:09 AMselect count(distinct(<column_name>)) from <table>
column_name % 64 is by which producer decides the kafka partition to send the data intoJackie
06/30/2020, 2:11 AMJackie
06/30/2020, 2:12 AMPradeep
06/30/2020, 2:14 AMPradeep
06/30/2020, 2:14 AMPradeep
06/30/2020, 2:14 AMJackie
06/30/2020, 2:15 AMPradeep
06/30/2020, 2:16 AMPradeep
06/30/2020, 2:16 AMJackie
06/30/2020, 2:18 AMJackie
06/30/2020, 2:18 AMJackie
06/30/2020, 2:19 AMPradeep
06/30/2020, 2:19 AMPradeep
06/30/2020, 2:19 AMPradeep
06/30/2020, 2:28 AMCaused by: java.lang.OutOfMemoryError: Direct buffer memory
Pradeep
06/30/2020, 2:31 AMPradeep
06/30/2020, 2:36 AMKishore G
Neha Pawar
Pradeep
06/30/2020, 4:52 PM