Hey all looking for some guidance on an EKS-to-EK...
# general
n
Hey all looking for some guidance on an EKS-to-EKS Pulsar cluster migration and hoping someone here has been through something similar. Context We're migrating a production Pulsar cluster to a new EKS cluster in the same region. Both clusters run Pulsar 4.0.x deployed via Helm. We have a handful of high-throughput topics with persistent subscriptions that we need to carry over cleanly. Approach we've been evaluating Our primary candidate is geo-replication via
set-clusters
on namespaces, combined with
replicateSubscriptionState(true)
on the consumer side to sync subscription cursors to the destination. This gets us near-zero downtime: producers keep running, replication backfills the destination, and consumers cut over once backfill is complete. The problem we keep running into is that
replicateSubscriptionState
only syncs
markDeletePosition
-
individuallyDeletedMessages
does not transfer. As covered in StreamNative article, for subscriptions with any out-of-order acking (which ours have, due to concurrent processing), the redelivery window on the destination can be significant. Our application is not idempotent - double-processing messages causes real issues downstream, so we can't just absorb the duplicates and move on. Alternative we've considered Drain the backlog before cutover - stop producers, let consumers fully drain so
individuallyDeletedMessages
collapses and
markDeletePosition
catches up, then cut over. This would give us a clean cursor state. The downside is that stopping producers means measurable downtime, and with the volume we're dealing with, the drain window could be long. We'd rather avoid that if there's a better path. What we're looking for Has anyone found a clean way to handle this that avoids both the duplicate redelivery problem and extended producer downtime? Open to anything different migration strategy, tooling, a Pulsar feature we've missed, or just a recommendation on how to scope the drain window to make it acceptable. Any experience here would be really helpful.
🎯 1
l
> The problem we keep running into is that
replicateSubscriptionState
only syncs
markDeletePosition
-
individuallyDeletedMessages
does not transfer. Yes, this is true. However, there has been improvements to the existing solution where it was possible that subscription replication didn't sync the markDeletePosition in all cases. explained in https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1772824988082859 > Our application is not idempotent - double-processing messages causes real issues downstream, so we can't just absorb the duplicates and move on. This is a problem since Pulsar is at-least-once and there will be duplicates in certain cases in normal operations with a single cluster. Applications should be idempotent or tolerate duplicates in a way or another. Pulsar Transactions doesn't fix this issue since it ties messaging operations to be atomic. If a transaction fails, messages get redelivered, and from the application's perspective they are duplicates. In Pulsar, it's possible to avoid duplicates on the producer side when a producer retries sending a single message. This feature is disabled by default. If that feature is disabled, producers can produce duplicate messages that are stored in Pulsar. > Drain the backlog before cutover - stop producers, let consumers fully drain so
individuallyDeletedMessages
collapses and
markDeletePosition
catches up, then cut over. This would give us a clean cursor state. The downside is that stopping producers means measurable downtime, and with the volume we're dealing with, the drain window could be long. We'd rather avoid that if there's a better path. This seems to be a reasonable alternative. There's also a feature in Pulsar that helps reducing the cut off time. It's the "blue-green cluster deployment" (Pulsar 3.0 release blog , PIP-188 document). Unfortunately there's no documentation in the reference doc about this feature (contributions are welcome!)
👀 1
🙏 1
y
Appreciate the response thankyou!