@RunLLM is the consumer name in ConsumerOpts (golang) used for consumer message routing in Key_Shared subscription?

No, the consumer name in ConsumerOptions (Go) is not used for message routing in Key_Shared subscriptions. Message routing in Key_Shared subscriptions is determined by the message key or ordering key, not the consumer name. The consumer name is only an identifier for the consumer instance and does not affect how messages are distributed among consumers in a Key_Shared subscription type. For Key_Shared, messages with the same key are always routed to the same consumer, regardless of the consumer name, as described in the official documentation and Go client source code (Pulsar Go client, Key_Shared subscription, consumer.go). Would you like more detail on how message routing works in Key_Shared subscriptions? Sources: - Pulsar Go client - Key_Shared subscription - consumer.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM I have two topics, one subscription name, and i want the messages (across both topics) with the same keys to go to the same consumers E.g. message A from topic A and message B from topic B with the same partition key should go to the same consumer Is that possible by setting consumer name?

No, this is not possible by setting the consumer name. In Pulsar, message routing for Key_Shared subscriptions is based on the message key (or ordering key) and the internal hash mapping algorithm, not the consumer name. The mapping of keys to consumers is done independently for each topic, so there is no guarantee that messages with the same key from different topics will be routed to the same consumer, even if the consumers have the same name and share a subscription (Pulsar concepts-messaging, Key_Shared subscription). Would you like more detail on how Pulsar's Key_Shared routing works or possible workarounds? Sources: - Key_Shared subscription 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM isn't the internal routing based on some consumer identifier that maps consumers to partition ranges?

No, the internal routing for Key_Shared subscriptions is not based on a consumer identifier. Instead, it uses a hash of the message key (or ordering key) to assign messages to consumers. The mapping is managed by the broker using algorithms like auto-split hash range or consistent hashing, and the assignment is per topic, not globally across topics. The consumer name is not used for routing; it is only an identifier (Pulsar concepts-messaging, Key_Shared subscription). Would you like more detail on the specific algorithms used for key-to-consumer mapping? Sources: - Key_Shared subscription 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how is a consumer mapped to a auto-split hash range?

In Pulsar's Key_Shared subscription with auto-split hash range, each consumer is assigned a region within the hash range 0 to 65,535. When a new consumer connects, the largest existing region is split in half, and the new consumer is assigned one half. The mapping is managed by the broker, and a message's key is hashed and mapped to a region; the consumer responsible for that region receives the message (Pulsar concepts-messaging: auto-split hash range). Would you like a step-by-step example of how the regions are split and assigned? Sources: - Pulsar concepts-messaging: auto-split hash range 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yep I would like a detailed explanation of how a consumer is assigned an auto-split hash range

Certainly! Here’s a step-by-step explanation of how a consumer is assigned an auto-split hash range in Pulsar’s Key_Shared subscription mode: 1. Initial State: The full hash range (0 to 65,535) is assigned to the first consumer. 2. Adding Consumers: When a new consumer connects, the broker finds the largest existing region and splits it in half. The lower half is assigned to the new consumer, and the upper half remains with the original consumer. 3. Subsequent Joins: Each time a new consumer joins, the largest region (across all consumers) is split, and the new consumer gets the lower half. 4. Message Routing: For each message, the broker hashes the message key, computes

hash % 65536

, and finds which region (and thus which consumer) the result falls into. Example: - 1st consumer (C1): gets [0, 65536) - 2nd consumer (C2): [0, 32768) (C2), [32768, 65536) (C1) - 3rd consumer (C3): [0, 16384) (C3), [16384, 32768) (C2), [32768, 65536) (C1) - 4th consumer (C4): [0, 16384) (C3), [16384, 32768) (C2), [32768, 49152) (C4), [49152, 65536) (C1) When a consumer disconnects, its region is merged with the adjacent region on the right. For more, see Pulsar concepts-messaging: auto-split hash range. Sources: - Pulsar concepts-messaging: auto-split hash range 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM as a result, no property of the consumer is going to influence which hash region it's assigned? only the time it joins and what the broker assigns the consumer?

Correct. In auto-split hash range mode, no property of the consumer (such as consumer name or ID) influences which hash region it is assigned. The assignment is determined solely by the order in which consumers join and the broker's region-splitting logic (Pulsar concepts-messaging: auto-split hash range). Would you like more detail on how the broker manages these assignments? Sources: - Pulsar concepts-messaging: auto-split hash range 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM but must consumer name be unique for the subscription?

No, the consumer name does not have to be unique for a subscription. Pulsar allows multiple consumers with the same consumer name to attach to the same subscription, especially for Shared and Key_Shared types (Pulsar concepts-messaging: subscriptions). However, using non-unique consumer names can lead to operational confusion and, in some cases, issues with features like DLQ producers or monitoring (Pulsar Go client issue #1297). Would you like more detail on the implications of using duplicate consumer names? Sources: - Pulsar concepts-messaging: subscriptions - Pulsar Go client issue #1297 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM I'm using a string org_id as my msg ordering key with keybased batching and key_shared subscription but my two consumers are seeing messages from all org_ids, why is that?

If both consumers are seeing messages from all org_ids, the most likely cause is that either batching is not set to key-based batching, or both consumers are not using the same subscription name. For Key_Shared to guarantee that all messages with the same key (org_id) go to the same consumer, you must use key-based batching on the producer and ensure both consumers are on the same subscription (Key_Shared subscription, Batching for Key_Shared). Would you like a checklist or code example to verify your setup? Sources: - Key_Shared subscription - Batching for Key_Shared 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM and the msg ordering key is used to route messages to the right consumers?

Yes, in a Key_Shared subscription, the message ordering key (or key) is used to route messages to the correct consumer. All messages with the same key or ordering key are always delivered to the same consumer, preserving order for that key (Key_Shared subscription). Would you like more detail on how the routing and ordering work? Sources: - Key_Shared subscription 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM which takes more precedence, key or ordering key?

The ordering key takes precedence over the key. If an ordering key is set on a message, it is used for routing and message dispatch in Key_Shared subscriptions; if not, the key is used (TypedMessageBuilder.java, Key_Shared subscription). Would you like more detail or code examples? Sources: - TypedMessageBuilder.java - Key_Shared subscription 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM I'm also using sticky policy, [32768, 65535] and [0, 32767] and seeing key overlap in my two consumers, how come?

If you are using the sticky policy with explicit hash ranges (0, 32767] and [32768, 65535]) and still see key overlap between your two consumers, it is likely that the hash ranges are not being set correctly or there is an overlap/misconfiguration in the consumer setup. Each consumer must be assigned a non-overlapping, explicit range, and the broker will reject consumers if their ranges overlap ([Pulsar concepts-messaging: sticky, KeySharedPolicySticky Go client). Please verify that: - Each consumer is assigned only its intended range. - The ranges do not overlap and together cover [0, 65535]. - No consumer is connecting without specifying a range. Would you like a code example for correct sticky policy usage? Sources: - Pulsar concepts-messaging: sticky - KeySharedPolicySticky Go client 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.