Hi team. I noticed the sample config for segmentPartitionConfig in Pinot. But I have a question about this. If highlevel type is used, is it ok to use a different partition config in Pinot from kafka partition config? eg, set numPartitions a value not the same as Kafka topic partition number. Or if no partition key is set in Kakfa, will it take effect in Pinot if a column is used in columnPartitionMap?

Are you planning on using HLC? It is an old non-scalabe solution that will be deprecated

Yes. The reason I’m trying to use HLC is Kafka partition is not based on the dimension that is used in most queries. Trying to configure Pinot table to partition segments based on this dimension.

Yeah, you want to use LLC for sure, as it is a lot more scalable. As for matching partitioning, do you have a choice to repartition upstream? If not, even with sorting + replica-group, you can get a lot of scalability.

When LLC is used, is it ok to use the following config, which is not the same partition policy as Kafka topic partition?