I see sure I can use Murmur.. I also tried to call the rebalance endpoint with

minAvailableReplicas: 0

but I now get the following message

Instance reassigned, table is already balanced

Ok I see the confusion. Basically, when you are configuring the partitioning, you are telling Pinot that the upstream data is already partitioned as per your specifications. Rebalancer won’t do the partitioning for you.

Ohh I see so I only have 2 shards in my kinesis which is partitioned by accountId.So that mean my config has to look like this.Secondly I dont know what algorithm kinesis is using so can I enable partitioning on my realtime table ?

"segmentPartitionConfig": {
      "columnPartitionMap": {
        "accountId": {
          "functionName": "Murmur",
          "numPartitions": 2
        }
      }

You’ll have to ensure that Kinesis is partitioning the events using the same implementation of murmur as Pinot. @Kartik Khare is there a way to specify this on kinesis side?

We dont have that much throughput now and we have recently increased our servers to 3 but reason why we wanted to add partitioning is because none of our queries are eventTimestamp bound so we wanted to avoid querying all segments.I thought having partitioning might improve performance since we are expected to have a high volume of events soon

Depending on your workload they can be.

thanks