Hi, my team are arguing about if pinot table partition config depends on kafka topic partition policy. We’re using lowlevel type to consume kafka stream data. According to this doc(the first picture), it doesn’t depend on kafka partition policy. But according to another doc(the second picture), it does depend on kafka partition policy. Could somebody help make it clear?

It does depend on Kafka partition function which is typically murmur hash

I’m using RealtimeToOfflineSegmentsTask in my realtime table config. So is there any suggestion on configuring Offline table to use replica group instance assignment or other method that could improve query performance?

What’s the qps

currently 100, expect much higher

Then partitioning plus sorting within each segment will definitely help

+1

I’m not very clear what does ‘partitioning plus sorting’ mean.😅

So let’s say you have a dimension in your schema which appears in all/most queries like

where userId = xxx

, then it is likely a good candidate to sort/partition your data on that dimension.

yes, we have such a dimension, like userId, and used in most queries

Yes then that is the one to sort and partition on

I think I can use sort index in my case. Due to some reason, upstream partition function is not implemented based on this dimension.

Partition definitely helps if you want to do thousands of read qps.

pretty low, 100 currently. Expect query performance improved to allow higher qps.

What’s the final target qps and p99 latency you need

Not very sure now. Just try to figure out how to tune query performance with limited resources and what I could do to achieve it.🤣

Ok.