Andrei
03/07/2026, 7:50 PMAndrei
03/07/2026, 7:51 PMNaor Hadad
03/11/2026, 9:26 AMset-clusters on namespaces, combined with replicateSubscriptionState(true) on the consumer side to sync subscription cursors to the destination. This gets us near-zero downtime: producers keep running, replication backfills the destination, and consumers cut over once backfill is complete.
The problem we keep running into is that replicateSubscriptionState only syncs markDeletePosition - individuallyDeletedMessages does not transfer. As covered in StreamNative article, for subscriptions with any out-of-order acking (which ours have, due to concurrent processing), the redelivery window on the destination can be significant. Our application is not idempotent - double-processing messages causes real issues downstream, so we can't just absorb the duplicates and move on.
Alternative we've considered
Drain the backlog before cutover - stop producers, let consumers fully drain so individuallyDeletedMessages collapses and markDeletePosition catches up, then cut over. This would give us a clean cursor state. The downside is that stopping producers means measurable downtime, and with the volume we're dealing with, the drain window could be long. We'd rather avoid that if there's a better path.
What we're looking for
Has anyone found a clean way to handle this that avoids both the duplicate redelivery problem and extended producer downtime? Open to anything different migration strategy, tooling, a Pulsar feature we've missed, or just a recommendation on how to scope the drain window to make it acceptable. Any experience here would be really helpful.Amanda
03/11/2026, 4:46 PMKP
03/17/2026, 4:18 AMFabri
03/17/2026, 6:39 PMPartha Talukdar
03/18/2026, 5:14 PMreason: "Failed to add or retrieve schema."Partha Talukdar
03/18/2026, 5:17 PMLari Hotari
03/23/2026, 7:24 PMGrégory (Kosmos)
03/24/2026, 9:43 AMconsumer.negativeAcknowledge(msg);
• consumer.reconsumeLater(msg, 1000, TimeUnit.MILLISECONDS);
Is it just the possibility to customize the delay ?Amelia Mink
03/30/2026, 4:38 AMAmjad Ali
03/31/2026, 7:18 AMLari Hotari
03/31/2026, 11:20 PMAmanda
04/09/2026, 9:25 PMpulsar-talos-toolset-0:/pulsar$ bin/pulsar-admin namespaces get-clusters geo-replication/testing
pulsar-okd1
pulsar-talos
Replicated subscriptions enabled:
pulsar-talos-toolset-0:/pulsar$ cat conf/broker.conf | grep -i replicatedSu
enableReplicatedSubscriptions=true
Configured Auto Failover Client:
ServiceUrlProvider failover = AutoClusterFailover.builder()
.primary(primaryUrl)
//secondary can be a list of multiple back up clusters - good to know for when we deploy a 3+ cluster environment in the future
.secondary(List.of(secondaryUrl))
.failoverDelay(30, TimeUnit.SECONDS)
.switchBackDelay(60, TimeUnit.SECONDS)
.checkInterval(1000, TimeUnit.MILLISECONDS)
.build();
PulsarClient pulsarClient = PulsarClient.builder()
.serviceUrlProvider(failover)
.build();
Configure Producer to use failover client and produce messages to a fresh topic (geo-replication/testing/test):
public SyslogProducer(PulsarClient client) throws PulsarClientException {
this.client = client;
this.producer = client.newProducer(Schema.STRING)
.topic("persistent://" + CFG.getString("pulsar_tenant") + "/" + CFG.getString("pulsar_namespace") + "/"
+ CFG.getString("pulsar_topic"))
.create();
}
producer = new SyslogProducer(new TalosFailoverClient().getAutoFailoverClient());
Configure Producer to use failover client with replicated subscriptions and consume messages on fresh topic and subscription:
public SyslogConsumer(PulsarClient client) throws PulsarClientException {
this.client = client;
this.consumer = client.newConsumer(Schema.STRING)
.topic("persistent://" + CFG.getString("pulsar_tenant") + "/" + CFG.getString("pulsar_namespace") + "/"
+ CFG.getString("pulsar_topic"))
.subscriptionName(CFG.getString("pulsar_subscription"))
.replicateSubscriptionState(true)
.subscribe();
}
consumer = new SyslogConsumer(new TalosFailoverClient().getAutoFailoverClient());
Tested the following:
• started producer on talos cluster on auto created test topic
• topic replicated successfully to okd1 cluster (can see it in topics list on okd1 cluster)
• started consumer for the first time on talos cluster after 600+ msgs were sent
• messages started consuming at message value 660
where did all of the other messages go?
• topic stats on talos:
0 msgs in all backlog values
replicated connected state = true
• topic stats on okd1:
backlog size: 120731, replication backlog: 982, replication connected state = false
• shutdown/uninstalled talos cluster
• producer stopped producing messages with connection error
auto cluster failover failed since it did not switch to okd1 cluster and continue producing
• started/reinstalled talos cluster:
producer never started again
Attempting Test 1 again:
Using new topic and subscription (still using geo-replication/testing tenant/ns with geo-rep enabled). All same configurations
• started producer on talos cluster test1 topic
• topic replicated to okd1 successfully
• immediately checked topic stats on okd1:
okd1 replication connection originally was true with no msgs in the replication backlog or backlog size
checked topic stats again shortly after and replication connection switched to false with messages growing in replication backlog and backlog size
• checked topic stats on talos:
• backlog size is 0 with no consumer even started
shouldn’t backlog be growing since no consumer is receiving the msgs?
• replication connection = true, backlog size is 0
• started consumer on talos cluster on test 1 topic:
• started receiving msgs at msg value 500
continued message loss
replication backlog continues to grow on okd1
• stopped consumer on talos test 1 topic, continued producer
• started consumer again on talos test 1 topic
this time it picked up where it left off - no msg loss
• shut down/uninstalled talos cluster:
producer and consumer failed to switch to okd1
• restarted/installed talos cluster:
producer failed to reconnect and continueShasank Pandey
04/14/2026, 11:03 AMLari Hotari
04/16/2026, 12:04 PMAmanda
04/16/2026, 5:54 PMJoão Fernandes
04/17/2026, 12:36 PMreader.GetLastMessageID() in golang)
Is it guaranteed that I'll get the latest message ID in that topic? Or will I need to publish a sentinel message and wait to read to guarantee that I'm truly seeing the topic's tail?
Thank you!Shasank Pandey
04/18/2026, 8:06 AMLari Hotari
04/23/2026, 2:58 PMLari Hotari
04/28/2026, 2:40 PMLari Hotari
04/29/2026, 2:06 PMLari Hotari
04/30/2026, 7:22 PMSlackbot
05/08/2026, 9:58 AM萧易客
05/14/2026, 7:05 AMAmanda
05/21/2026, 7:13 PMDaniel Kaminski
05/26/2026, 2:32 PMFabri
05/27/2026, 6:05 AMGrégory (Kosmos)
05/28/2026, 5:07 PMdocker run -it -p 6650:6650 -p 8080:8080 -e TZ=Europe/Paris --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:3.0.17 bin/pulsar standalone --advertised-address localhost
Then I have these 3 java files, I run SimpleConsumerWithRetry, and gets logs :
>> pulsar client created
>> pulsar DLQ consumer created
>> pulsar consumer created
Then I run SimpleProducer, here the logs
>> pulsar client created
>> pulsar producer created
deliver msg 9:1:-1, value:Message de 2026-05-28T19:00:43.731+02
Then I look at the log of SimpleConsumerWithRetry :
2026-05-28T19:00:43.798+02: receive msg 9:1:-1, value:Message de 2026-05-28T19:00:43.731+02
2026-05-28T19:00:45.902+02: receive msg 18:0:-1, value:Message de 2026-05-28T19:00:43.731+02
2026-05-28T19:00:48.899+02: receive msg 18:1:-1, value:Message de 2026-05-28T19:00:43.731+02
So far so good : 2 retries, with 2 seconds delay.
But the message is not send to the DLQ (at least, not the DLQ topic I'm listening).
If I ask the topic list curl --silent -XGET <http://localhost:8080/admin/v2/persistent/public/testns> there is only 2 topics (I was expected 3 : main, retry, dlq) : ["<persistent://public/testns/test>","<persistent://public/testns/test-subs-DLQ>"] .
I tried also to not listen to test-subs-DLQ (removing one consumer in the code) and there is only 1 topic created (test). Any clue ?Grégory (Kosmos)
05/29/2026, 3:42 PMreconsumeLater. When I set delayTime to around 1 second (or less), the message is redelivered immediately. I wrote a small integration test to reproduce this behavior (the test requires a valid Docker installation to start a Pulsar container).
When running the TestReconsumeLater.java test, if the constant REDELIVER_DELAY is set to less than 1010 ms, the test fails because the message is redelivered in 34 ms instead of the expected delay:
java.lang.AssertionError:
Expected actual:
34L
to be greater than or equal to:
1009L
Is this a known limitation ? Is it documented anywhere ?