Hi! Can anyone provide feedback to the following failed test in our Pulsar environment: Test 1: Pulsar Kubernetes Clusters with Global Configuration Store. Pulsar v4.0.9 Helm Chart v4.5.0 Geo-replication enabled on tenant/ns geo-replication/testing:

pulsar-talos-toolset-0:/pulsar$ bin/pulsar-admin namespaces get-clusters geo-replication/testing
pulsar-okd1
pulsar-talos

Replicated subscriptions enabled:

pulsar-talos-toolset-0:/pulsar$ cat conf/broker.conf | grep -i replicatedSu
enableReplicatedSubscriptions=true

Configured Auto Failover Client:

ServiceUrlProvider failover = AutoClusterFailover.builder()
        .primary(primaryUrl)
        //secondary can be a list of multiple back up clusters - good to know for when we deploy a 3+ cluster environment in the future
        .secondary(List.of(secondaryUrl))
        .failoverDelay(30, TimeUnit.SECONDS)
        .switchBackDelay(60, TimeUnit.SECONDS)
        .checkInterval(1000, TimeUnit.MILLISECONDS)
        .build();

    PulsarClient pulsarClient = PulsarClient.builder()
        .serviceUrlProvider(failover)
        .build();

Configure Producer to use failover client and produce messages to a fresh topic (geo-replication/testing/test):

public SyslogProducer(PulsarClient client) throws PulsarClientException {
        this.client = client;
        this.producer = client.newProducer(Schema.STRING)
                .topic("persistent://" + CFG.getString("pulsar_tenant") + "/" + CFG.getString("pulsar_namespace") + "/"
                        + CFG.getString("pulsar_topic"))
                .create();
    }
producer = new SyslogProducer(new TalosFailoverClient().getAutoFailoverClient());

Configure Producer to use failover client with replicated subscriptions and consume messages on fresh topic and subscription:

public SyslogConsumer(PulsarClient client) throws PulsarClientException {
        this.client = client;
        this.consumer = client.newConsumer(Schema.STRING)
                .topic("persistent://" + CFG.getString("pulsar_tenant") + "/" + CFG.getString("pulsar_namespace") + "/"
                        + CFG.getString("pulsar_topic"))
                .subscriptionName(CFG.getString("pulsar_subscription"))
                .replicateSubscriptionState(true)
                .subscribe();
    }
consumer = new SyslogConsumer(new TalosFailoverClient().getAutoFailoverClient());

Tested the following: • started producer on talos cluster on auto created test topic • topic replicated successfully to okd1 cluster (can see it in topics list on okd1 cluster) • started consumer for the first time on talos cluster after 600+ msgs were sent • messages started consuming at message value 660 where did all of the other messages go? • topic stats on talos: 0 msgs in all backlog values replicated connected state = true • topic stats on okd1: backlog size: 120731, replication backlog: 982, replication connected state = false • shutdown/uninstalled talos cluster • producer stopped producing messages with connection error auto cluster failover failed since it did not switch to okd1 cluster and continue producing • started/reinstalled talos cluster: producer never started again Attempting Test 1 again: Using new topic and subscription (still using geo-replication/testing tenant/ns with geo-rep enabled). All same configurations • started producer on talos cluster test1 topic • topic replicated to okd1 successfully • immediately checked topic stats on okd1: okd1 replication connection originally was true with no msgs in the replication backlog or backlog size checked topic stats again shortly after and replication connection switched to false with messages growing in replication backlog and backlog size • checked topic stats on talos: • backlog size is 0 with no consumer even started shouldn’t backlog be growing since no consumer is receiving the msgs? • replication connection = true, backlog size is 0 • started consumer on talos cluster on test 1 topic: • started receiving msgs at msg value 500 continued message loss replication backlog continues to grow on okd1 • stopped consumer on talos test 1 topic, continued producer • started consumer again on talos test 1 topic this time it picked up where it left off - no msg loss • shut down/uninstalled talos cluster: producer and consumer failed to switch to okd1 • restarted/installed talos cluster: producer failed to reconnect and continue

I would review this - I know @Lari Hotari improved the documentation as there can be some race conditions with replicated subscription so the recomendation is to pre-configure the topics with geo-replication. Check some of the limitations too