@RunLLM After upgrading from 2.11 to 4.0 Pulsar, some namespaces disabling geo-replication to other cluster. Why and how to fix this? For auth using ssl certs. Traffic go between pulsar-proxy

After upgrading from Pulsar 2.11 to 4.0, namespaces may lose geo-replication if the allowed clusters or replicated clusters settings are not properly migrated or re-applied. This can also occur if there are changes in how cluster metadata, authentication, or TLS settings are handled between versions. In particular, issues with SSL/TLS configuration or missing cluster permissions can prevent replication from being enabled, especially when traffic flows through pulsar-proxy. According to Pulsar 3.2.x geo-replication docs, both the tenant and namespace must explicitly allow and set the correct clusters for replication, and the proxy/broker must have valid SSL certs and trust configuration. To fix this, verify that: 1. The tenant's

allowed-clusters

and the namespace's

replicated-clusters

are set to include all intended clusters using

pulsar-admin tenants update

and

pulsar-admin namespaces set-clusters

. 2. All clusters' metadata is updated and reachable, and SSL certs are valid and trusted by all brokers and proxies (see example). 3. If using Kubernetes or StreamNative Operator, ensure CRDs like

replicatedClusters

are set in the namespace spec (see StreamNative Operator issue). Would you like more detail on troubleshooting SSL/TLS or on the admin commands to re-enable geo-replication? Sources: - Pulsar geo-replication docs - see example - see StreamNative Operator issue 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM replication work after restarting broker of unloading namespace. but in few minutes, I see info message, that replicator disconected from one side and Producer closed from other. No any other info

If replication resumes after a broker restart or namespace unload but then the replicator disconnects and the producer closes shortly after, this typically indicates a transient connectivity or configuration issue. Common causes include network interruptions, authentication/authorization mismatches, or a misconfiguration in cluster or namespace replication settings. The info-level log messages about replicator disconnects and producer closures are expected when the broker intentionally closes these connections, but the absence of error or warning logs suggests the broker considers this a normal or expected event. To diagnose further, check: - That all clusters in the replication set are reachable and have valid SSL/TLS and authentication settings (see Pulsar geo-replication docs). - That the namespace’s replication clusters are correctly set and consistent across all clusters. - For any subtle issues in the broker logs at debug level, as info messages alone may not reveal the root cause (see related issue). Would you like more detail on how to enable debug logging or on specific admin commands to check replication status? Sources: - see Pulsar geo-replication docs 👍 0 button 👎 1 button 💡 Hint: Mention @RunLLM in the thread for followups.

@Stanislaw Makarawec Which Pulsar version exactly? One of the fixed bugs related to Proxy is https://github.com/apache/pulsar/pull/24181 which could impact geo-replication.

4.0.6

Do you get any error messages?

I have 2 problems. First one, is that stand-by cluster disconnect replication after 1 minute. Not all topics, but some of them.

does replication work at all, or is it just the disconnection problem that exists? if it doesn't work at all, perhaps it's an authorization issue? could you check that there's

produce

permission for the role that handles replication?

After unload replication can work 1-2 minutes and then again disconected

how many CPU cores do your brokers have?

Ok, thats good questions. We set cpu watching on usage. for Bookie 1, broker: 3 from one side and Bookie 1, broker: r: 1 L; 3

one thing that could help is to collect a thread dump with

jstack 1

in the broker container just before the disconnection happens. To see if there's a deadlock. https://jstack.review/ is a great tool for analysing the thread dump. In any case, please file a GitHub issue about your problem at https://github.com/apache/pulsar/issues .

Maybe you know about some limitation for replication in one not partitioned topic. We have topic with 12k messages per sec, but replication can process only 7-7,5k messages. This topics was placed on broker without any others topics(bundles). It's some limitation of resources or configuration?

We have topic with 12k messages per sec, but replication can process only 7-7,5k messages. This topics was placed on broker without any others topics(bundles). It's some limitation of resources or configuration?

Is this a regression with Pulsar 4.x or did the same problem exist before? The size of the messages matter most and whether they are batched since Pulsar will be replicating the original batches (also called "entries" in Pulsar) created by the producer clients. BookKeeper performance matters on the target cluster side so that it could write the replicated batches fast enough. A large BookKeeper journal disk helps get IOPS fast and if you can sacrifice durability, it's possible to disable fsync in BookKeeper (

journalSyncData=false

) to go faster when the journal disk isn't fast. In cloud disks, you usually get a relatively low amount of IOPS if your disk is small. To play safe, increasing

journalMaxGroupWaitMSec

from the default value to

1

to

2

and increasing

journalBufferedWritesThreshold

helps reducing number of IO operations.

12k we get only on new version. We doesn't have this load on previous, so we can't compare. Our bookie has nmve disks. But only one per bookie. (ledger and journal placed on the same disk)

Although placing ledger and journal on the same disk isn't recommended, with local nvme disks, I wouldn't assume it to be a bottleneck at this scale. local nvme disks are over 100x faster than small cloud disks.

@Lari Hotari hello. After long time, we found the issue with geo-replication to stand-by cluster. With active-active replication, where the second cluster was stand-by (without active consumers and active subscriptions), replication stopped after one minute. The logs only showed an info message: Disconnect replicator at position 1682677:116895 with backlog 0. After that, the replication backlog started to accumulate on the stand-by cluster side. The issue was resolved by disabling automatic topic deletion, specifically: brokerDeleteInactiveTopicsEnabled=false.

Since don't have any active subscriptions, do you have retention configured in the stand-by cluster for the namespace?

Any retention doesn't configured. On 2.x (2.8, 2.11) it's work normaly, but after update to 4.X we faced with this. Now we enabled subscription replication (excluding delayed topics) and now it's work normal (with brokerDeleteInactiveTopicsEnabled=false). Previously, some topics were disabled at the moment of replication and were not reconnected, even with replicated subscriptions.