This message was deleted.
# generals
Slackbot
01/10/2023, 8:56 PMThis message was deleted.
t
Tim Corbett
01/10/2023, 9:02 PMSpecifically, we find the existing hash ring or hash range splitting behavior to produce lopsided throughput to consumers, especially in our environment where consumers autoscale frequently to meet demand. Our consumers are not enhanced by maintaining a constant relationship between specific keys and individual consumers, so simply tracking which hashes are currently outstanding for which consumers, and routing any new hashes to the next available/least loaded consumer would suffice.
Tim Corbett
01/10/2023, 9:04 PMAdditionally, I don't see why the choice of routing algorithm (and perhaps even parameters) could not be part of the namespace/topic policies instead of global. There are mixed use case workloads where the consistent hash routing would still benefit speed/memory complexity.
Tim Corbett
01/10/2023, 9:05 PMOthers have mentioned this issue in the past as well, such as https://github.com/apache/pulsar/issues/15705
d
David K
01/10/2023, 9:12 PMI would suggest cross posting this in the #C5ZSVEN4E channel as well. Having said that, is there an existing PIP that this is associated with? Do you have an outstanding PR for you changes?
t
Tim Corbett
01/10/2023, 9:13 PMI do not believe there is a PIP yet but I haven't checked exhaustively. I have written a bit of code but do not want to get too far without a way forward to merge potentially.
Tim Corbett
01/10/2023, 9:13 PMIt would be wise when moving forward to get more input on the desired architecture as well, of course. I'm still new to this codebase.
d
David K
01/10/2023, 9:16 PMUnderstood. I think the folks on the #C5ZSVEN4E channel will have some good advice for getting a feature request added. PIPs are generally reserved for larger architectural changes to the platform so depending on what you are suggesting, e.g. making the routing algorithm pluggable, it might just be an enhancement vs. a PIP
t
Tim Corbett
01/10/2023, 9:16 PMAhh, thanks for that distinction.
👍 1