@RunLLM There is an issue with replication. Geo-replication is enabled. The publish rate to the topic is about 20 msg/s. At some point, the producer starts publishing at a rate of 5k msg/s for about two minutes. During that time, replication stops and a replication backlog begins to accumulate. After the publish rate returns to the normal value of around 20 msg/s, replication does not recover. How can this be fixed?

This issue is likely related to Pulsar's replication rate limiting or backlog quota policies. When a sudden spike in publish rate (e.g., 5k msg/s) occurs, the replication backlog can exceed configured limits, causing replication to pause or stop. If the backlog or rate limits are not reset or adjusted, replication may not automatically recover even after the publish rate drops back to normal. To fix this, check and increase the replication backlog quota and replication rate limits for the affected topic or namespace, and ensure the broker is not throttling replication due to exceeded quotas. Also, verify that the replication connections are healthy and not blocked by other resource constraints. For more, see ReplicatorStats and dispatch/publish throttling settings. Would you like more detail on how to check or adjust these settings? Sources: - ReplicatorStats - dispatch/publish throttling settings 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM i don't get any log message about disabling replication. Only by metrics i get info about problems with replication. Do you know something about same problemsa?

Replication issues may not always generate explicit log messages about replication being disabled; often, problems are only visible through metrics such as replication backlog or disconnected status. This is a known observability gap—current Pulsar admin APIs and logs may not provide clear, actionable replication error messages, and users often rely on metrics to detect issues. There is an open discussion about improving geo-replication observability, such as splitting out replication-specific status from general topic stats for easier monitoring (Improving Geo-Replication Observability). Would you like more detail on how to monitor or troubleshoot replication issues using metrics? Sources: - Improving Geo-Replication Observability 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how dispatchThrottlingRatePerReplicatorInMsg is working?

dispatchThrottlingRatePerReplicatorInMsg

sets the maximum number of messages per second that each replicator can dispatch in Pulsar. If set to 0, throttling is disabled. This limit helps control replication traffic between clusters and prevent resource exhaustion. The setting is applied per replicator, and can be configured in

broker.conf

or dynamically via admin APIs (ServiceConfiguration.java). Would you like more detail on how to configure or monitor this setting? Sources: - ServiceConfiguration.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM does this param help me in situations with publish spikes?

Yes,

dispatchThrottlingRatePerReplicatorInMsg

can help during publish spikes by limiting the replication message rate, preventing a single spike from overwhelming the replication channel or remote cluster. However, if the publish spike exceeds this limit, a replication backlog will accumulate, and replication will proceed at the throttled rate until the backlog is cleared (ServiceConfiguration.java). Would you like more detail on how to tune this parameter for your use case? Sources: - ServiceConfiguration.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@Lari Hotari hello. Maybe you know any information with such problem? If no, i will create issue and try to reproduce with pulsar-standalone.

@Stanislaw Makarawec which Pulsar version?

4.0,7 latest LTS

I'd recommend taking a few thread dumps and a heap dump once that happens. Broker thread dumps don't contain sensitive data unless a an IP address for Zookeeper is considered sensitive (I think Zookeeper threads have IP in the name). Sharing a heap dump isn't recommended since it contains sensitive broker data. For a kubernetes cluster you could use this script to automate triggering the thread dumps and heap dump and downloading them with kubectl: https://github.com/lhotari/pulsar-contributor-toolbox/blob/master/scripts/collect_jvm_diagnostics_from_pod.sh . It creates 3 thread dumps 3 seconds apart and a single heap dump on the first round. Various tools can be used to inspect the state of Pulsar broker internals from the heap dump and that usually helps. In addition, it would be useful to get topic stats and topic stats internal too since those might also bring up some useful information.

Thank you. When I return to this task I will try to reproduce this problem on demo cluster and will create issue with dumps

just don't share the heap dump publicly, for demo data you could encrypt with gpg with the recipient's key (which can be found with an email address in most cases)