Good morning! I have a question regarding geo-replication: I have a two cluster geo-replication set up deployed via kubernetes. when running a producer on cluster A for the first time (using auto topic creation - did not manually create it) my producer successfully runs and sends the message on the cluster A service url. before starting the consumer on cluster B on that same topic, I wanted to check the topic stats on cluster B, but it says the topic doesn't exist. Once I start the consumer, the consumer on the cluster B service url, it receives the messages successfully. But I thought topics were supposed to be replicated across clusters? Is this an error in my geo-replication set-up, or is this expected behavior? Would enabling replicated subscriptions help this issue?

“Once I start the consumer, the consumer on the cluster B service url, it receives the messages successfully.” It sounds like geo-replication is working on this namespace, why do you consider this an error?

the topic does not exist on cluster B until I connect the cluster B consumer. shouldn't the topic on cluster A be replicated to cluster B? I would expect to be able to view the topic stats on cluster B and see the messages sitting in the backlog before starting that consumer. Instead I get the "topic does not exist" error on cluster B.

When you connect a consumer on cluster B, have all the messages been replicated, or only those published to cluster A after connecting the consumer? I want to understand when the topic is created on cluster B vs. when it is available in the metadata via pulsar-stats, etc.

when I connect the cluster B consumer, it receives all of the messages from cluster A producer. until I start that cluster B consumer, the topic does not exist on cluster B and none of the messages exist on cluster B.

I would characterize that as the topic exists, meaning the messages are stored inside BookKeeper. However, the metadata store isn’t updated with the new topic information due to replication, and the metadata is only updated by attaching a consumer. This sounds like a bug.

Okay I thought this may be a bug as well. I am not sure if it is a timing issue. I believe I've waited a few minutes before connecting the consumer, but would like to verify. I will run a few more tests to confirm how long it takes (if it is a timing issue) and see if the messages exist in cluster B bookkeeper. an error we get in the cluster A broker logs on immediate producer send is:

2025-09-09T22:20:32,547+0000 [broker-client-shared-scheduled-executor-7-1] WARN  org.apache.pulsar.client.impl.PulsarClientImpl - [topic: <persistent://geo-replication-2/testing/__change_events>] Could not get connection while getPartitionedTopicMetadata -- Will try again in 754 ms                                                                                                                                                                                                            │
│ pulsar-talos-broker 2025-09-09T22:20:32,551+0000 [pulsar-io-3-15] ERROR org.apache.pulsar.client.impl.ClientCnx - [id: 0x714b44aa, L:/ - R:] Close connection because received internal-server error {"errorMsg":"","reqId":1946041700241505712, "remote":"pulsar-okd1-broker.pulsar.svc.cluster.local/, "local":"/"}                                                                                                                │
│ pulsar-talos-broker 2025-09-09T22:20:32,552+0000 [pulsar-io-3-15] WARN  org.apache.pulsar.client.impl.BinaryProtoLookupService - [<persistent://geo-replication-2/testing/__change_events>] failed to get Partitioned metadata : {"errorMsg":"{"errorMsg":"","reqId":1946041700241505712, "remote":"pulsar-okd1-broker.pulsar.svc.cluster.local/", "local":"/"}","reqId":1229027975051488438, "remote":"", "local":"/"}                 │
│ pulsar-talos-broker org.apache.pulsar.client.api.PulsarClientException$LookupException: {"errorMsg":"{"errorMsg":"","reqId":1946041700241505712, "remote":"pulsar-okd1-broker.pulsar.svc.cluster.local/1"local":"/"}","reqId":1229027975051488438, "remote":"", "local":"/"}

I assumed this could also be a timing issue because it immediately tries to find the topic on cluster B (okd1) and it does not exist. I'll get back to you soon with those results!

Check the logs on cluster B for a ’create topic” command, so we can confirm the time it was created.

@David K testing update: It's been about 25 minutes and still the topic does not exist on cluster B (okd1). I am not seeing anything in the okd1 broker logs for creating a topic, and I am not seeing any evidence in the cluster B bookies that the messages exist on cluster B. On cluster A (talos) I was able to successfully send 100 msgs from the talos producer. On send I immediately get that error I sent you in my previous message. The topic does exist on the talos cluster, and I can see the topic stats on the talos cluster:

bin/pulsar-admin topics stats <persistent://geo-replication/testing/test>
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "bytesInCounter" : 4490,
  "msgInCounter" : 100,
  "systemTopicBytesInCounter" : 0,
  "bytesOutCounter" : 0,
  "msgOutCounter" : 0,
  "bytesOutInternalCounter" : 0,
  "averageMsgSize" : 0.0,
  "msgChunkPublished" : false,
  "storageSize" : 4490,
  "backlogSize" : 4490,
  "backlogQuotaLimitSize" : -1,
  "backlogQuotaLimitTime" : -1,
  "oldestBacklogMessageAgeSeconds" : -1,
  "publishRateLimitedTimes" : 0,
  "earliestMsgPublishTimeInBacklogs" : 0,
  "offloadedStorageSize" : 0,
  "lastOffloadLedgerId" : 0,
  "lastOffloadSuccessTimeStamp" : 0,
  "lastOffloadFailureTimeStamp" : 0,
  "ongoingTxnCount" : 0,
  "abortedTxnCount" : 0,
  "committedTxnCount" : 0,
  "publishers" : [ ],
  "waitingPublishers" : 0,
  "subscriptions" : { },
  "replication" : {
    "pulsar-okd1" : {
      "msgRateIn" : 0.0,
      "msgInCount" : 0,
      "msgThroughputIn" : 0.0,
      "bytesInCount" : 0,
      "msgRateOut" : 0.0,
      "msgOutCount" : 0,
      "msgThroughputOut" : 0.0,
      "bytesOutCount" : 0,
      "msgRateExpired" : 0.0,
      "replicationBacklog" : 100,
      "connected" : false,
      "replicationDelayInSeconds" : 0,
      "msgExpiredCount" : 0
    }
  },
  "deduplicationStatus" : "Disabled",
  "nonContiguousDeletedMessagesRanges" : 0,
  "nonContiguousDeletedMessagesRangesSerializedSize" : 0,
  "delayedMessageIndexSizeInBytes" : 0,
  "compaction" : {
    "lastCompactionRemovedEventCount" : 0,
    "lastCompactionSucceedTimestamp" : 0,
    "lastCompactionFailedTimestamp" : 0,
    "lastCompactionDurationTimeInMills" : 0
  },
  "ownerBroker" : "pulsar-talos-broker-2.pulsar-talos-broker.pulsar.svc.cluster.local:8080"

This section of the stats indicates no data was replicated from the talos cluster.

I did pass in the right service url for okd1 by doing a clusters update on the okd1 cluster and updated the urls to use the load balancer IP address. Then I restarted the brokers. (did this for both clusters). I am not using any authentication credentials for either of the clusters. I do have permissions to create a topic on that namespace. verified by just doing a topic create command on the ns.

Maybe you can change the logging level to “DEBUG” on okd1 and try again. That way we can see what calls are made to the cluster if any.

@David K still hitting the same issue. I did enable debug on both clusters. there are no create topic logs, but the debug logs on okd1 show the metadata lookup for the topic created on talos cluster:

2025-10-02T23:16:33,265+0000 [pulsar-io-3-5] DEBUG org.apache.pulsar.broker.service.BrokerService - No autoTopicCreateOverride policy found for <persistent://geo-replication/testing/test>
2025-10-02T23:16:33,471+0000 [pulsar-io-3-8] DEBUG org.apache.pulsar.broker.service.ServerCnx - [<persistent://geo-replication/testing/test>] Received PartitionMetadataLookup from /10.128.2.45:43198 for 770175375804561621
2025-10-02T23:16:33,471+0000 [pulsar-io-3-8] DEBUG

still hitting the same internal server error in the logs on cluster A as well on producer send. the topic stats in cluster A show that the replicator to ok1 exists, but it says false:

"replication" : {
    "pulsar-okd1" : {
      "msgRateIn" : 0.0,
      "msgInCount" : 0,
      "msgThroughputIn" : 0.0,
      "bytesInCount" : 0,
      "msgRateOut" : 0.0,
      "msgOutCount" : 0,
      "msgThroughputOut" : 0.0,
      "bytesOutCount" : 0,
      "msgRateExpired" : 0.0,
      "replicationBacklog" : 100,
      "connected" : false,
      "replicationDelayInSeconds" : 0,
      "msgExpiredCount" : 0
    }

Is this expected behavior for replication? I guess it would make sense that the topic wouldnt be created on cluster B unless there was a reason for it to be (like when the consumer is ready to receive messages). Therefore, the replicator wouldn't connect until the topic is created on cluster B

Thank you for running the test. It sounds like the metadata on cluster B doesn’t get updated until a consumer attaches to the replicated topic even though the underlying topic data is there.

right, I think so. would this be considered a bug I should further look into? or is geo-replication working as expected?

This behavior is wrong, and the topic should exist in the target cluster’s metadata. Can you create an issue in the Apache Pulsar Github repo for this? Thanks!

Issue created: https://github.com/apache/pulsar/issues/24819 Thank you for the help! 🙂

Thanks for continuing to test this. The geo-replication behavior of a bare metal deployment and a K8s deployment should be the same.

@David K I am still working on resolving this issue, and I have documented the following findings: I believe I may have made progress on narrowing down the issue, which is cross communication across clusters for replicators. As we know, geo-replication is semi-working: issue re-cap: • 2 pulsar kubernetes cluster set-up (okd1 and talos) • updated cluster urls to load balancer IPs (manually) and restarted brokers with success • can use the cluster urls successfully with producers and consumers • enabled geo-replication on replication tenant/ns • produced 100 messages on talos cluster with no consumers running on either cluster • immediately triggers connection/internal-server error in talos broker logs • topic never exists on okd1 cluster • started consumer on okd1 cluster and it receives the 100 messages result: geo-replication semi-working with weird topic replication issue. The topic never replicates, even though on consumer start, the messages replicate. theory: by default and behind the scenes for geo-replication, the talos replicator communicates with the okd1 broker NOT the LB/Proxy. Hence why the error log shows the connection error happening with the internal okd1 broker dns name. But we can’t do cross-cluster-communication with brokers because they do not expose an external endpoint, only internal. It is trying to connect over the internal pod network of okd, which won’t work. What we have been using is a proxy service in kubernetes that sits behind a loadbalancer which has an external IP, and this is the only external IP we are using in our kubernetes cluster set up. I am trying to look into pulsar configurations that would resolve this, but I am not sure if it is because of our kubernetes set up. configurations in broker.conf that may help resolve the issue: • advertisedAddress ◦ I changed advertisedAddress to our LB IP in broker.configData in our helm chart, and it breaks the deployment, the pods aren't able to come up. • advertisedListeners • internalListenerName • bindAddress • createTopicToRemoteClusterForReplication ◦ this is set to true for our brokers, so this shouldn't be the issue. Any feedback you may have is appreciated!

sorry, I haven’t had much time to look at this.

@David K no worries, still doing some testing but no luck yet 😕. Will keep you updated if I make any progress