This message was deleted.

Thanks! So, if I have two messages with the same subscription key hash in different topics (due to a different routing key or due to addition of a new partition), then it's possible that two consumers will be processing messages for the same key? (one consumer is reading from the P1, and the seconds from P2)

Yes, if you are using two different topics with a different number of partitions that would be the case. However, in that scenario, you can use the sticky policy of the KS subscription to ensure that consumers of the two different topics are consuming the same set of keys.

Sorry, perhaps, my explanation was unclear( I do not consume two topics, I consume one topic, but with two partitions (P1 and P2). Let's say I have a message M1 in P1 and a message M2 in P2. Then I create a KS subscription. M1 and M2 have the same message key, so in KS they should be processed in order. I'm trying to understand in what order. Will it be M1->M2 based on some logic (maybe sequence ID...) or the order is not guaranteed here because I have two separate subscribers for each topic?

Thanks for the clarification. Short answer is that the processing order will be non-deterministic. If you need total global ordering then you would have to implement the logic yourself using your own set of consumers (one per partition), and the publishTime property of each message to determine which message is next.

Got it) Or, perhaps I can use the same key for both partition routing and SK subscription key. Unfortunately, it seems I won't be able to scale my topic anyway, so I'm kinda limited to the predefined number of partitions as in Kafka 😔 Thank you anyway, this was very helpful)

“I can use the same key for both partition routing and SK subscription key”, yes this is the preferred approach. Why can’t you scale your topic? Pulsar has no limitation on the partition count.

Hmm. Does Pulsar do some kind of rebalancing when adding a new partition? Let's say I have two partitions, one contains messages for key

K1

and the second - for

K2

P1: K1, K1...
P2: K2, K2...

Then I add new partition, so new messages with the same hash can be delivered to the new partition (as I understand). So, now we'll have:

P1: K1, K1...
P2: K2, K2...
P3: K1, K1...

Pulsar does do rebalancing of key ranges when partitions are added/removed. By default it will pause all consumers on a KS subscription and allow the consumers to complete the processing of the previously delivered messages before reassigning the consumers to the partitions.

That's what I'm talking about. The consumers will finish processing of the delivered messages, but not all messages in the partition. So it's possible to have unprocessed messages with

K1

in one old partition, and new messages with

K1

in the newly added partition

I think the KS subscription handles that scenario transparently for you.

I hope so) But you know, as they say, "trust but verify")