Hi! What is the partitioning function used in Pinot for

murmur

? Upstream Kafka uses

murmur2

. Does Pinot expect

murmur2

or

murmur3

? Appreciate any guidance 🙇‍♀️

Murmur in Pinot is murmur2. It also supports murmur3

Good to know! Thanks @Mayank! So by default Pinot would match Kafka’s murmur2 partitioning. We’d just have to ensure the partition column/message key and number of partitions matches with upstream. What are the steps in case we want Pinot to use murmur3?

You can set murmur3 in table config. Also, always check committed segments metadata to ensure it only has 1 partition id

@Mayank Does Pinot support multi-column partitioning to align with composite key partitioning in Kafka?

For upsert you can have composite key, yes

@Mayank Our scenario is a non-upsert real-time Pinot table being configured for partition-based segment pruning. In that case, we can configure only one column - is it?

Do you mean like a composite key? If so, then yes.

I meant to confirm if composite key i.e. partition-based segment pruning using multiple columns is possible in Pinot ONLY for upsert-based real-time tables and not supported otherwise?

Note multiple columns is different from composite columns for partitioning. Multiple columns is supported (but I don't think I know anyone who has used it), composite columns are not.

Ah got it! I thought partition-based segment pruning in Pinot when configured for multiple columns would function as a combined key, as that's how we'd be able to set it up in upstream Kafka.

Yes. One column + time based pruning will already prune out a lot

Makes sense thanks