Hey guys, we were running pulsar 2.11 and after upgrade to 3.0.0, all tests fail, we get errors like "topic not found" or "namespace not found". Did something change? Are topics/namespaces setup differently now?

One possible reason could be a known problem that metadata changes are eventually consistent to some level in Pulsar. If a test creates a namespace or topic and immediately attempts to access it, it might result in the problem that you are describing. Unfortunately, the work around is to add a short delay or add polling to ensure that the creation has completed before proceeding in the test. This known issue has been well described in this issue comment: https://github.com/apache/pulsar/issues/12555#issuecomment-955748744 It would be helpful if you could isolate a reproducer. It could be files in a GitHub gist or a separate repo that reproduces the problem in isolation.

The test runs in a single node, so it should always be consistent, right?

The test runs in a single node, so it should always be consistent, right?

That's a fair assumption, but it might apply also to the single node case.

No, it's the default pulsar image, and I've seen failure from Java client when creating consumer, and Python client, when creating producer.

ok, this happens when Pulsar isn't yet fully started.

No, not using test containers

this is how Testcontainers handles it: https://github.com/testcontainers/testcontainers-java/blob/eadc9d6ec7777562d3b335b[…]rc/main/java/org/testcontainers/containers/PulsarContainer.java

Yes, I had a loop trying to create a test topic and this exception is now not being caught by that loop. So it seems before we would return some other exception in this case, and now we are return TopicNotFound (which I was not handling). Let me add that to the catch statement and see if that works

If you don't use functions and/or statestore, it's worth disabling those in the container to reduce resource consumption and ensure faster startup. Memory parameters aren't very good either. Those might be causing issues with 3.x since ZGC is the default.

thanks

The memory issue can be handled by passing PULSAR_MEM=-Xmx512M and PULSAR_GC=-XX:+UseG1GC environment to the container.

yep, catching the TopicNotFound in my ready loop solved it... thanks! :)

ah yep, I remember this

but I assume it's also enabled by default in production?

which deployment option do you use? Function worker is disabled by default in broker.conf , but enabled by default for Pulsar standalone.

ok, then it's already off by default

Another tip for standalone is to select to use Zookeeper instead of the Rocksdb based metadata store. https://github.com/apache/pulsar/discussions/20985#discussioncomment-8951010