Can anyone tell me the expected behavior for repli...
# general
a
Can anyone tell me the expected behavior for replication topic stats in a 2 cluster set up with a global metadata store cluster? • manually created topic and subscription using pulsar-admin on cluster A ◦ topic then existed on cluster B because of the global metadata cluster, but the sub did not since that does not belong to global metadata • Run Producer and consumer (using replicated subs) on cluster A - sending and receiving messages successfully on cluster A ◦ view topic stats on cluster A: replication backlog is 0 and replication connected state = true ▪︎ subscription stats show isReplicated false, replicated false ◦ view topic stats on cluster B: replication backlog is growing and replication connected state = false ▪︎ subscription stats show nothing - sub does not exist on cluster B • Stop consumer on cluster A, continue producer on cluster A, start consumer on cluster B, consumer is successfully receiving messages but messages were lost from where consumer A left off and where consumer B picked up ▪︎ view topic stats on cluster A: replication backlog is 0 and replication connected state = true • subscription stats show isReplicated false, replicated false ▪︎ view topic stats on cluster B: replication backlog is growing and replication connected state = false • subscription stats show isReplicated false, replicated false Should the replication state of the remote cluster be false if I am not actively producing messages on that cluster? Am I doing something wrong with replicated subscriptions? Is auto/lazy topic/sub creation causing issues here? Do I need to manually create the topic and sub on both clusters before running producers and consumers? When I manually created the topics and subs, the connected status was true on the remote cluster, and sub replication was working with true stats as well.
l
> subscription stats show isReplicated false, replicated false Please check https://pulsar.apache.org/docs/4.2.x/administration-geo/#enable-replicated-subscription . It's necessary to enable
replicateSubscriptionState
in the client for consumers when creating the consumer. Alternatively you can use
pulsar-admin topics set-replicated-subscription-status --enable
to enable it. There's a proposal to improve this so that subscription replication could be controlled with policies at the namespace level for all subscriptions, but that hasn't been implemented. The current solution requires changes in clients or by explicitly changing the subscription status..
There are some limitations in replicated subscriptions to take into account:
• Replicated subscriptions use periodic snapshots to establish a consistent association between message positions across clusters. Snapshots are taken every
replicatedSubscriptionsSnapshotFrequencyMillis
milliseconds (default: 1000 ms), but the effective granularity of mark-delete position updates on the remote cluster depends on how many messages are produced between snapshots. See Replicated subscriptions snapshot configuration and tuning for details.
• Only the mark-delete position (the baseline cursor position) is replicated — individual acknowledgments are not. Messages acknowledged out of order may be redelivered after a cluster failover.
• Replicated subscriptions do not provide consistent behavior when consumers are active on multiple clusters simultaneously. Most messages will be processed on both clusters (duplicate processing), and some may be processed on either cluster depending on replication timing. To avoid this, process messages on a single cluster at a time when using replicated subscriptions.
• Delayed message delivery impairs subscription replication. The mark-delete position does not advance until delayed messages have been delivered and acknowledged, so replication lags behind accordingly.
I used Claude Desktop with Deepwiki MCP and this prompt:
Copy code
check apache/pulsar and apache/pulsar-site to find an answer to this question using deepwiki mcp:
<your question>
The AI answer:
A few things are getting conflated here. Short version: messages aren't really being lost, and the "connected=false" on cluster B is actually expected. The subscription just isn't replicated even though you created it, so cluster B's consumer starts from
Latest
on a lazily-created local sub.
Direction of the
replication
map in topic stats
The
replication
section shows outbound replicators from the cluster you're querying — not inbound. So on cluster B,
replication["A"]
is B→A, not A→B. Since you're not producing on B, that replicator has nothing to do, so
connected=false
is correct and expected. A's
replication["B"]
showing
connected=true, backlog=0
is the actual A→B replicator working fine.
Short answer to "should replication state on the remote cluster be false if I'm not producing there?" — yes.
Why
isReplicated=false
on the subscription
pulsar-admin topics create-subscription
does not create a replicated subscription — there's no
--replicated
flag on create. A subscription becomes replicated only when: • a consumer connects with
.replicateSubscriptionState(true)
, or • you explicitly run
pulsar-admin topics set-replicated-subscription-status --enable
Also requires
enableReplicatedSubscriptions=true
on every broker (default true, worth confirming).
If stats show
isReplicated=false
, the feature is not on — no snapshots, no cursor sync.
Topic on B but no sub on B
Expected. Topics live in the configuration store (global). Subscriptions are local per-cluster state. That part of your setup is correct.
What actually happened on failover
Because the sub wasn't actually replicated:
A produces → messages replicate to B and pile up as backlog (normal, no consumer on B yet)
A's consumer acks → acks are not propagated to B (no snapshot markers being written)
1. You stop A's consumer, start B's consumer on the same sub name → sub doesn't exist on B, gets lazily auto-created at
Latest
2. B starts consuming from "now", the gap looks like loss
Not loss — just a fresh sub starting at Latest with no idea where A's cursor was.
To make failover actually work
1.
enableReplicatedSubscriptions=true
on all brokers
replicationClusters
on the namespace includes both A and B
Create the subscription on both clusters before producing (lazy creation isn't safe here — snapshots can only tie cursors together if the sub exists on both sides when snapshots are written). Use
pulsar-admin topics create-subscription
at
Earliest
on both.
Every consumer on both clusters must use
.replicateSubscriptionState(true)
— or run
set-replicated-subscription-status --enable
on both clusters
1. Verify with
pulsar-admin topics stats
that
subscriptions.<sub>.isReplicated=true
on both clusters before you trust failover
Heads up: replicated subs are eventually consistent, bounded by snapshot frequency (1s default) plus network RTT. The feature is designed to prevent loss, not duplicates — you should expect some duplicates around failover. If you're seeing loss and no duplicates, the feature isn't active.
Quick check
Copy code
pulsar-admin topics get-replicated-subscription-status <topic> -s <sub>
Run on both clusters. If it's not enabled on both, that's your answer.
Refs: PIP-33 for the design,
ReplicatedSubscriptionTest
in pulsar-broker for a working end-to-end example.
One point that the above AI answer missed is that the subscription will get automatically created on the remove cluster unless it has already been created.
I hope this helps making progress on your side!
a
I wil give these configurations a try, thank you so much for the feedback!