@RunLLM I have many log messages with org.apache.pulsar.broker.service.persistent.PersistentReplicator Skip the reading due to new detected schema How ti avoid this?

This log message means Pulsar's geo-replication is pausing message replication because a new schema version was detected and the replicator must fetch and validate the schema before continuing. This is normal behavior to ensure schema compatibility and message order, but frequent schema changes can cause repeated pauses and impact replication throughput. To reduce these pauses, minimize schema updates on replicated topics and avoid frequent incompatible schema changes. For more, see PersistentReplicator.java and GeoPersistentReplicator.java. Would you like more detail on how schema changes trigger this or how to tune schema compatibility settings? Sources: - PersistentReplicator.java - GeoPersistentReplicator.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how to minimize schema updates?

To minimize schema updates in Pulsar, avoid making frequent changes to your message schema (such as adding, removing, or modifying fields) on topics that are geo-replicated. Use stable, versioned schemas and batch schema changes when possible. This reduces the number of schema versions and limits replication pauses caused by schema validation. For more, see the schema compatibility check documentation. Would you like more detail on best practices for schema evolution or how to enforce schema stability? Sources: - Schema compatibility check 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@Stanislaw Makarawec There might be a bug too. I remember seeing this log line when branch-4.0 was broken a while ago in https://github.com/apache/pulsar/issues/24670 . Besides cherry-picking PIP-428 to branch-4.0, I guess https://github.com/apache/pulsar/pull/24680 was a fix by @Yubiao Feng to address the issue, but I could be wrong. Do you happen to use topic policies? (just thinking of the PIP-428 impact for your case)

It's only for one way replication? If so, i need to fix two way replication 🙂 Now i will disable one way and try to reproduce disconnections with two ways

well it might reproduce in two way. replication in pulsar is essentially one way. In two way replication you just have both ways configured.

@RunLLM after disabling geo-replication, messages still persist

After disabling geo-replication, messages may still persist because Pulsar only deletes a geo-replicated topic when all producers/consumers are disconnected, all subscriptions are deleted, and retention is not set. Simply disabling replication does not immediately remove messages or topics; you must also ensure there are no active subscriptions and retention is disabled. For geo-replicated topics, deletion relies on automatic garbage collection, not manual removal. See Pulsar geo-replication topic deletion. Would you like more detail on how to force topic or message deletion? Sources: - Pulsar geo-replication topic deletion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM log messages.

If you are referring to log messages related to geo-replication (such as schema fetch pauses or replication state changes), these are normal in Pulsar and indicate internal replication or schema management events. They do not mean messages are deleted or lost; logs are for monitoring and debugging. If you want to reduce log verbosity, adjust the broker log level settings. Would you like more detail on how to configure log levels or interpret specific log messages? Sources: - https://github.com/apache/pulsar/pull/17154 - https://github.com/apache/pulsar/issues/24380 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@Lari Hotari hello. One interesting thing. When I disable geo-replication, that log messages still exist. Disable it by setting only local cluster for namespaces. Why I still get this messages?

Perhaps the replicator just didn't stop cleanly. I saw something similar in https://apache-pulsar.slack.com/archives/C078TGY9R29/p1757495401962059?thread_ts=1757485289.077439&cid=C078TGY9R29 when I tried debugging it. Please report an issue at https://github.com/apache/pulsar/issues to track it

Can help brokers restart or not? Namespace unload doesn't help 😞

Yes, please report an issue. I saw a similar problem where the replicator was stuck in stopping as explained before. The issue might be related to a fix https://github.com/apache/pulsar/pull/24680, as explained previously. That links to https://github.com/apache/pulsar/pull/24225 which is addressed in Pulsar 4.1.0. The PR 24225 fix might give ideas what could be causing the issue.