Using 3.2.1 Pulsar QQ #1: Does

Failover

,

Exclusive

,

Shared

,

Key_Shared

Subscriptions work on all the partitions in a topic or they are limited to only one partition in a topic ? When using OpenMessagingBenchmark with Failover mode, we see that there is delay up on a partitioned topic. Wondering whats the behaviour of these subscription modes on a partitioned topic QQ #2: We are creating a

tenant (c1, c2, c3 .. cN, cAgg)

,

namespace (c1, c2, c3,. ... cN, cAgg)

,

topic (c1, c2, c3 ... CN, cAgg)

and using per message replication clusters to

(cAgg)

this gives us flexibility to aggregate messages from all (1..N) clusters to (cAgg). Any scalability concerns to be worried about when N (number of clusters) are in few thousands ?

very good questions. Please create a discussion at https://github.com/apache/pulsar/discussions/new?category=q-a with these questions and then promote it here.

The short answer is that subscriptions work across all partitions in the topic. Except for the failover subscription, which assigns a consumer to a partition (like the Kafka consumer group model).

I am hesitated to answer questions like those in slack. Before you ask question, show what study/research/document you read. This way, we can understand which part is missing for users to self-help and improve the project.

@Yu Wei Sung Sure there can be discussions, but there's a tradeoff: your effort to answer the question on Slack will be mostly lost after a while. If you instead ask the person who asked the question to ask it on GitHub discussions, that will build up a valuable knowledge base for the Apache Pulsar community over time.

Though Q1 looks silly off the OMB + Pulsar docs, there are some practical observations we had. Lemme explain further. We have 5 standalone clusters (s1..5) & 1 distributed pulsar (d1) & geo-replication is enabled between these 6 clusters. we have twewaked OMB to support running producers on one side (s1..s5) and running consumers on other side (d1). The profile is 10K messages/sec (10MB/s/producer) for producers (total 50K messages/sec, 50MB/sec). With Failover subscription mode on the 1 consumer, we did see that lag kept increasing on a partitioned topic with 10 partitions. https://pulsar.apache.org/docs/next/client-libraries-consumers/#subscribe-to-topics talks about 1 consumer per partition to be created on a partitioned topic. So, the complexity is with finding the right math for this tuple (topic, # partitions, subscriptionName, subscriptionType, #consumerObjects). It isn't very clear for a topic with N partitions with one microservice running with a Subscription of type, how many consumer objects to be created to ensure that the consumer can keep up with the # producers ?

Pulsar has the shared subscription type that supports the competing consumers pattern

Wasn't aware of this new Q/A forum. Posting the questions from there : https://github.com/apache/pulsar/discussions/22437 & https://github.com/apache/pulsar/discussions/22438

Thank you @Naresh for the great questions! We'll reply on https://github.com/apache/pulsar/discussions/22437 & https://github.com/apache/pulsar/discussions/22438