Hi, guys. As described in <https://pulsar.apache....
# general
h
Hi, guys. As described in https://pulsar.apache.org/docs/next/concepts-replication/#aggregation-replication, there is an aggregation replication mode. Yet in the doc of https://pulsar.apache.org/docs/next/administration-geo/, there is no instruction of setting the aggregation mode. So my question is how to set the aggregation replication mode ? Thank you.
l
there is no special "aggregration replication mode". The documentation is not very clear and doesn't provide an example of such configuration.
The "aggregation replication" pattern referred in the docs is about configuring the replication in a hub-spoke way where instead of having point-to-point connections (full mesh) between each cluster, you have a hub cluster and you connect all other clusters to this hub.
I think it's just confusing to call it "aggregration replication".
h
Per my understanding, normal replication provides two-way ability, how one-way (aggregation) can be done via configuration ?
I’ve tried to create two pulsar clusters without shared metadata store (zookeeper), the replication seems ok (one-way), yet the metadata between clusters (eg. topic partitions) must be done by myself.
l
I have been using geo-replication without a shared metadata store. At least in that case, replication is configured point-to-point for each one-way direction. Yes, there are some gotchas with partitioned topics. I think one solution is to disable topic auto creation. I believe that the replicator will attempt to create the topic in the remote cluster when replication starts the first time. In addition, setting up authorization and TLS isn't very straightforward.
h
Agreed. It seems I have done exactly the same thing. And I shall continue on that track. Thank you very much for your help.
👍 1
@Lari Hotari After some digging, I found out the
replicated subscription status
seems not working. Apart from that, the replication works well.
l
I found out the
replicated subscription status
seems not working.
@huangyanshuo what doesn't work there? Replicated subscriptions have some limitations. the discussion in https://github.com/apache/pulsar/discussions/22315 contains some details
h
I misunderstood the mechanism of replicated subscriptions. I originally thought that if I consume topic
t1
with subscription name
s
in cluster
z1
, the subscription
s
of topic
t1
in cluster
z2
won’t consume the same message and there would be a subscription created in the `z2`'s topic. An illustration would be:
Copy code
sub s1 -> topic t1 (in cluster z1)
sub s1 -> topic t1 (in cluster z2)
s1 in z1 & s1 in z2 shouldn't consume the same message
But after reading your excellent post, and I quote
The main limitation of replicated subscription is that only the "mark delete" position is replicated
, I think the behavior observed is quite normal and expected. Thank you again!
👍 1
thankyou 1