Hello. We are experiencing an issue with subscription replication. Pulsar version 4.0.7. We have two clusters: a primary production cluster and a secondary backup stand-by cluster. Subscription replication is enabled. At some point, a subscription stops replicating, and backlog begins to accumulate on the remote cluster (topic replication itself continues normally). Around a hundred topics are being replicated simultaneously, but the issue occurs only on one or a few of them. In some cases, the only workaround is to disable replication for the affected topic, clear the backlog on the remote cluster, and then re-enable replication. Sometimes a portion of updates still manages to replicate. There are no errors or warnings in the logs. What could be causing this issue? What additional information would be useful? In the topic stats for the subscription, the status shows replicated: true. Thanks!

possibly https://github.com/apache/pulsar/pull/24300 / https://github.com/apache/pulsar/issues/24380 related.

Pulsar allows you to set the message dispatch rate for all the replicators between replication clusters under a given namespace. The dispatch rate can be restricted by the number of messages per X seconds. Are these topics inside the same namespace? If so, can you share the output of the following?

pulsar-admin namespaces get-replicator-dispatch-rate <tenant>/<namespace>

@Lari Hotari this no, because this topics doesn't have delayed messages. About second, i don't know. @David K null Today I unloaded topic with problem subscription on main cluster and after that backlog disappeared on the remote cluster

Null means you are falling back to the cluster level settings l, which I believe is unthrottled

Looks like source cluster doesn't send updates for remote subscription

I'd assume that you are aware that subscription replication only supports replicating the so called mark-delete position. IIRC, the internal stats show the mark delete position. It's also possible that the conditions to send an update isn't fulfilled. https://gist.github.com/lhotari/96fda511a70d7de93744d868b4472b92 has a sequence diagram of the different steps that are involved before the subscription gets updated. One issue is that the snapshot cache might overflow. That's what https://github.com/apache/pulsar/pull/24300 is about. More explanation in https://github.com/apache/pulsar/issues/24380#issuecomment-2943211477 .

Thanks. I will try do it tomorrow or in a monday.

I'm sorry, it's called

pulsar_replicated_subscriptions_timedout_snapshots

increase(pulsar_replicated_subscriptions_timedout_snapshots_total[30s])

It was some problems