@RunLLM I am having a problem with topic replication. The topic replicates at about 10 messages per second, then for 2 minutes there is a sharp increase to 5k per second. Somewhere in the middle of the interval, replication to the remote cluster stops and does not resume. What should I do?

This issue may be related to replication backlog, network issues, or a bug in Pulsar's geo-replication logic. First, check the topic's replication stats using

pulsar-admin topics stats <topic>

to see if the

replicationBacklog

is growing or if the

connected

field is

false

for the remote cluster. Also, review broker logs for errors related to replication or producer failures, as these can indicate why replication stopped. If the backlog is large or the connection is lost, Pulsar may pause replication until the issue is resolved. If you see errors like "Failed to create remote producer" or "replicationBacklog" not decreasing, it could be due to a network partition, configuration mismatch, or a bug (see similar issues in Pulsar GitHub). Upgrading to the latest supported Pulsar version is recommended, as several geo-replication bugs have been fixed in recent releases. Would you like more detail on troubleshooting steps or configuration checks? Sources: - Pulsar GitHub Issue: Broker seeing high latency, high CPU and high GC Pauses - Pulsar GitHub Issue: Geo-replication doesn't create a topic and replicate messages before the topic exists in the remote cluster 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@Stanislaw Makarawec Which Pulsar version? Any error logs? https://github.com/apache/pulsar/pull/24189 fixed some issues

4.0.7. No errors. Looks like that

PR 24189 was a large change, so perhaps it has introduced a new issue. Have you checked topic stats and topic internal stats to see if the replication subscription or state shows anything

Yes. After that replication is disabled, and only set one cluster and return few cluster in replication fix it

@Stanislaw Makarawec It looks like a bug. Did you already file one? A thread dump could be useful to check if there's a dead lock. A heap dump would also be useful but it usually contains some of your data, so don't ever share heap dumps publicly.

Issue created: https://github.com/apache/pulsar/issues/25097 I was unable to reproduce bug with standalone 😞