Hi, we are using a Pulsar cluster of 3 nodes v3.1....
# generalj
Jerome B
02/22/2024, 4:02 PMHi, we are using a Pulsar cluster of 3 nodes v3.1.1. We're using partitioned topics. There are 3 hosts that run 25 instances of consumers configured in Key_Shared mode.
While I'm executing :
Copy code
sudo -u pulsar /opt/pulsar/apache-pulsar/bin/pulsar-admin topics stats <persistent://public/default/postprocess.request-partition-0> | grep keyHashRanges | grep -o "................$" | sort2147198192147483647Jerome B
02/22/2024, 4:03 PMWe're thinking this is linked to an issue we've got. We noticed that the key "D-25887" was very slow to consume.
We try to calculate murmur hash in python through this simple code (it needs the dependency mmh3==4.0.1 )
Copy code
import mmh3
print(mmh3.hash("D-25887".encode(), signed=False) & 0x7FFFFFFF)2147395553214719819Jerome B
02/22/2024, 4:45 PMConsistentHashing is enabled
l
Lari Hotari
02/22/2024, 6:56 PMHighest range value is(0x7FFBA4F3)214719819
Why it's notThe implementation explains this. https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apach[…]/broker/service/ConsistentHashingStickyKeyConsumerSelector.java Key ranges concept doesn't really map well to ConsistentHashingStickyKeyConsumerSelector. I don't think that there's a problem. In the implementation, each consumer gets mapped to the hash range 100 times. With this information it will select a consumer with this logic: https://github.com/apache/pulsar/blob/48c85ab1707004dc27022c494dbc557f7c3e888c/pul[…]/broker/service/ConsistentHashingStickyKeyConsumerSelector.java Therefore I think that the "keyHashRanges" is a misleading concept for this algorithm.(0x7FFFFFFF) ?2147483647
Lari Hotari
02/22/2024, 6:58 PMWe're thinking this is linked to an issue we've got. We noticed that the key "D-25887" was very slow to consume.It's possible that it's a coincidence that the hash of the key is close to integer maximum value. What client & version are you using ?
Lari Hotari
02/22/2024, 7:01 PMI spotted one bug here: https://github.com/apache/pulsar/blob/48c85ab1707004dc27022c494dbc557f7c3e888c/pul[…]/broker/service/ConsistentHashingStickyKeyConsumerSelector.java . The modulus operation isn't "sign safe". That matters when there's a hash key collision and the hash value is negative.
👍 1
Lari Hotari
02/22/2024, 7:01 PMHowever, I don't know if that applies in your case
Lari Hotari
02/22/2024, 7:02 PMI'd assume that there would be an index out of bounds exception in the logs
Lari Hotari
02/22/2024, 7:08 PMThe "sign safe modulus" fix: https://github.com/apache/pulsar/pull/22100
🙌 1
Lari Hotari
02/22/2024, 7:57 PMthe problem exists for other Key_Shared implementations too. In other implementation it seems to have a real impact.
j
Jerome B
02/22/2024, 10:27 PMThanks a lot for the detail, I'll spend some time understand the
selectConsistentHashingStickyKeyConsumerSelectorNavigateMapWhat client & version are you using ?We are using broker & client version 3.1.1
👍 1
3 Views