Hi We are testing segment partitioning for REALTIME tables K Apache Pinot #troubleshooting

Hi, We are testing segment partitioning for REALTI...

Surendra

05/25/2021, 5:05 PM

Hi, We are testing segment partitioning for REALTIME tables (Kafka as source) , But unable to find the configurations on documentation page except

When emitting an event to kafka, a user need to feed partitioning key and partition function for Kafka producer API

. Can someone give insights on how it works internally? How to configure schema registry for Kafka record keys ?

Neha Pawar

05/25/2021, 5:32 PM

does this help: https://docs.pinot.apache.org/basics/getting-started/frequent-questions/ingestion-faq#how-do-i-enable-partitioning-in-pinot-when-using-kafka-stream

Surendra

05/25/2021, 5:36 PM

Thanks Neha, Will check update you.

Surendra

05/25/2021, 11:24 PM

@Neha Pawar Ingested the data with the config mentioned in links. I see below info in metadata.properties file , is there any other way to validate partitioning is working as expected ?

Copy code

column.networkId.partitionFunction = Murmur
column.networkId.numPartitions = 2
column.networkId.partitionValues = 0,1

Neha Pawar

05/26/2021, 3:57 AM

Looking at the metadata is the right way to verify.

Neha Pawar

05/26/2021, 4:01 AM

But not sure why you are "0,1". Usually 1 segment consumes from 1 partition, and should only have one partition value

Neha Pawar

05/26/2021, 4:01 AM

Is the partitioning being done at Kafka producer side?

Surendra

05/26/2021, 4:02 AM

Yes, currently we have 2 partitions for kafka topic

Neha Pawar

05/26/2021, 4:11 AM

But did you set partitioner key as networkId on the producer?

Surendra

05/26/2021, 4:12 AM

Yes, we did

Surendra

05/26/2021, 4:12 AM

We have only one server in stage, does it creates any issues?

Open in Slack

Previous Next