https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
m
With fail-over subscription, there is one active consumer per partition
s
that still doesnt do the same as Kafka consumer group right? say I have 2 partitions and 2 consumers belong to same consumer group in kafka then if one of the consumer crashes the partition gets reassigned to other consumer which was also processing its own messages from its own partition (instead of doing nothing/standby)
how to achieve the same in Pulsar?
m
Yes, that’s how the failover already works
the partitions are assigned among the available consumers, 1 active consumer per partition
s
I see so I only have one consumer per partition (meaning no standby consumers per partition) then the behavior is same as kafka. is that right? Not sure how I missed this in the docs
d
Good callout @stephen fleming, we should update the docs to clarify this point.
s
There is lot more stress on the standby consumers (which I honestly think is not a scenario most people have) so it would be nice to know that if there are no standby consumers per partition and one of the consumer crashes other consumers can still take over. This is not highlighted at all
a
@stephen fleming I think you should find this information under this category in the docs: https://pulsar.apache.org/docs/3.2.x/concepts-messaging/#failover Hope this helps
s
Not clear enough to be honest, Like I said there is lot more stress on standby consumers. This one liner below can also be understood in many ways
Copy code
When the master consumer disconnects, all (non-acknowledged and subsequent) messages are delivered to the next consumer in line.
for example what is the next consumer here? is it a standby consumer or is it a consumer of another partition? Also, I dont think it is enough to just fix this one liner. somehow this section needs to be rewritten
m
@stephen fleming good point, it’s not very clear
to summarize failover subscriptions behavior: 1. Consumers connect to all partitions of a topic 2. Brokers, will pick one consumer to be active, all the others are just in stand-by 3. Consumer are picked in a way to spread the load evenly between them 4. When a consumer fails, brokers will replace it by promoting another standby consumer to be “active”
👌 1
s
Ahh that makes more sense!! may be then the confusion is in the word "standby" which gives a feeling that it doesn't do anything until it becomes active.
➕ 1
Copy code
Brokers, will pick one consumer to be active, all the others are just in stand-by
shouldn't this be
Copy code
Brokers, will pick one consumer per  partition to be active, all the others are just in stand-by
otherwise we are back to square one
m
Yes
✅ 1
The thing is that a “partitioned topic” in Pulsar is really just a collection of “simple topics”. The logic is then the same when applied to each partition
🙌 1
g
Sorry, I misread the type. In any case, it should be noted that any consumer going down changes the ownership for unaffected partitions as well. since partition-consumer mapping works on modulo and is not really sticky
a
@stephen fleming The long explanation after the line you quoted, doesn’t explain it all in detail? https://pulsar.apache.org/docs/3.2.x/concepts-messaging/#failover--partitioned-topics
@merlimat I have a follow up question. On the docs it is written : “If the number of partitions in a partitioned topic is greater than the number of consumers…” and later explains that the partitions are evenly divided among the consumers (say 9 partitions, 3 consumers). If the partitions are spread out (say even 1 partition per broker), how does the brokers coordinate dividing the load of partitions among the consumers?
g
all 3 consumer connect to all 9 partitions and the active consumer for each partition is chosen locally by sorting the consumer on its name and doing partition % consumer_count
🙏 1
Open in Slack
PreviousNext
Powered by