🚨 Unexpected Pulsar Outage: ZooKeeper Packet Size Exceeded Limit🚨 Hi everyone, we're running a production-grade Pulsar cluster and recently encountered a major outage. Our Pulsar broker abruptly failed to connect to the ZooKeeper node, causing the entire cluster to go down. Our Setup: • Broker & Proxy: Single EC2 instance (m7g.2xlarge) • BookKeeper: r7g.large (Journal: 40GB, Ledger: 150GB) • ZooKeeper: t4g.large What We Found: • The broker logs revealed the following error:

Br25-05-12T19:37:25,489+0000 [main-SendThread(ally-pulsar-zookeeper-0.prod.synaptic.services:2181)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x1000001f99e002f for server ally-pulsar-zookeeper-0.prod.synaptic.services/10.190.194.180:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.io.IOException: Packet len 10812855 is out of range!
        at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:124) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:84) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274) [org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
2025-05-12T19:37:25,590+0000 [main-EventThread] WARN  org.apache.pulsar.metadata.impl.ZKMetadataStore - Connection loss while executing batch operation of 1 GET_CHILDREN entries of total data size of 42. Retrying individual operations one-by-one.
2025-05-12T19:37:25,590+0000 [main-EventThread] WARN  org.apache.pulsar.metadata.impl.ZKMetadataStore - Connection loss while executing batch operation of 1 PUT entries of total data size of 230. Retrying individual operations one-by-one.

To mitigate the issue, we increased the

jute.maxbuffer

value from the default 10 MB (10485760) to 20 MB (20971520) (bin/pulsar file), which allowed the system to recover. Our Concern: We're trying to understand what caused the sudden spike in ZooKeeper packet size. Since ZooKeeper typically handles small metadata payloads, we're puzzled by this increase. One possible cause we suspect is the presence of numerous unused or stale topics, which may be bloating the metadata. We're now considering implementing a cleanup process to regularly delete old topics. Has anyone encountered a similar issue? Any insights on what might cause such a jump in ZooKeeper packet size or best practices to prevent this from happening again would be greatly appreciated. Thanks in advance!

Were there any recent increases in the number of topics, the data volume (leading to more entries in the managed ledger), or changes to the topic policies?

Here are some metrics that could help in observing some related details: https://github.com/apache/pulsar/blob/243ad5a22f7cc533e5a7382c01cb875dcdaee6bb/pul[…]/apache/pulsar/metadata/impl/stats/BatchMetadataStoreStats.java

pulsar_batch_metadata_store_executor_queue_size

,

pulsar_batch_metadata_store_queue_wait_time

,

pulsar_batch_metadata_store_batch_execute_time

and

pulsar_batch_metadata_store_batch_size

. Checking these metrics and finding some correlation with ZK or other Pulsar metrics could be helpful.

Hi Pulsar Team, I'm currently working on pulsar infrastructure sizing and limit. Could you help me to understand how port repartition is working between clients, proxy and brokers For instance, with 10 clients connected to 5 topics: how many inbound ports are opened on pulsar proxy machine? 10 or 50? Same question for brokers behind proxy? Does broker open same number of outbound ports? Finally: does pulsar mutualized open ports whatever the number of topics? Thanks.

message has been deleted

Hi all, Data Streaming Summit Virtual 2025 is happening on May 28–29. If you haven’t signed up already, please see the details below. It’s a free, virtual event with speakers from Netflix, Adobe, Uber, Databricks, and more focused on real-world messaging & streaming, AI infra, and lakehouse architectures. 📅 May 28 – 29, 2025 | 💻 100 % virtual 🎯 Theme: Unlocking Real-Time AI Why attend? ✅ Two action-packed days – 6 tracks, 40 + sessions ✅ Headline keynotes – Q6 Cyber, Alibaba, Netflix, Uber & StreamNative ✅ Free to attend – grab your complimentary pass in seconds Track highlights 1️⃣ & 2️⃣ Tech Deep Dives – Pulsar • Ursa • Kafka • Flink 3️⃣ Real-Time Analytics – dashboards, personalization, fraud detection 4️⃣ Streaming Lakehouse – Iceberg & Delta ingestion 5️⃣ AI & Agentic Systems – RAG pipelines, event-driven agents 6️⃣ Stream Processing – Flink, Spark, unified batch/stream 🔎 Explore the agenda: Full Agenda 🎟️ Claim your free pass: Registration Link Join thousands of data and AI builders as we unlock the future of real-time intelligence—see you online! 🙌

Hi team! We noted an issue in our production cluster. I found an earlier discussion that is relevant and I have added the details in the comment -- https://github.com/apache/pulsar/issues/5202#issuecomment-2882929294 Can someone take a look at this? Crux: The backlog reported for the topic was 30K+ whereas there was only 1 ledger. I see "Message expiry failed - mark delete failed" in broker logs during this period and the issue resolved on its own.

Hi team, I have a quick question, docs says recommended version of Java for pulsar > 2.10 is 17 here https://github.com/apache/pulsar/blob/master/README.md#pulsar-runtime-java-version-recommendation I'm currently using pulsar 3.2.3, will there be any compatibility issues if I use Java 21 as runtime? Should I have to upgrade my pulsar version to 4.0.4 which is latest? I have tried locally running pulsar 3.2.3 with java 21. I was able to start the standalone cluster and create topics etc

Hi 👋 1. I'm using the Go client

v0.15.0

and I wanted to ask if there a way to get messages from compacted topics other than using the

TableView

? I'm looking for a way to read messages from a stream (use of a channel) rather than storing all them at once in a

map[string]any

like the

TableView

does. It's for at least half a million records after compaction. 2. So I tried using a

Reader

with

ReadCompacted

option set to

true

but I'm getting all the messages: is there a way to actually read from the compacted topic with a

Reader

? (I'm using persistent topics) Thank you

Hi there 👋 I have set up Apache Pulsar and pushed 100 messages. While creating the namespace, I set subscriptionExpirationTime to 1 minute. On the consumer side, I've defined ConsumerType to be shared. After acknowledging all the messages, there are no subscribers, as they get removed based on my config. However, I'm wondering why space/storage is not freed up. As the docs say, once all the subscribers acknowledge the messages, it will be removed from the Pulsar system, so space should be freed. PS: Retention is already disabled. Can someone please help me understand how the messages will be removed from the broker and how space will be freed? Any help is appreciated.

hello everyone can someone help me about apachepulsar ? I got 8 server 3 zookeper 5 bookie and brokers and suddenly bookies die with no error ( I cant catch the error )

Pulsar site search on pulsar.apache.org is working again. It wasn't working for several months due to ASF Privacy policy changes (more details in GH discussion).

Hi there 👋 , I'm trying to create a topic using pulsar_perf and getting the following error. Can someone please help me fix it?

Topic creation encountered an exception by initialization topic policies service. org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException cluster-a

Hi there, I'm running the command, but the perf tool is not consuming the messages. Can anyone help me with this? Adding screenshots if this helps

pulsar-perf consume --service-url  <pulsar://localhost:6650> --subscription-type Shared --rate 10000 --num-subscriptions 10 --num-consumers 10 --subscription-position Earliest

Update: If I run

pulsar-perf consume --service-url  <pulsar://localhost:6650>

<persistent://public/default/my-topic>

then, it runs fine! Most likely, one of the parameters is causing the issue.

pulsarlogo 📣 [ANNOUNCE] Apache Pulsar 3.0.12, 3.3.7 and 4.0.5 released 📣 pulsarlogo For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: 3.0.12: https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.12/ 3.3.7: https://pulsar.apache.org/release-notes/versioned/pulsar-3.3.7/ 4.0.5: https://pulsar.apache.org/release-notes/versioned/pulsar-4.0.5/ Although 3.3.7 was released, please notice that support for 3.3.x has already ended and no new releases are planned.

Hi there, 👋 I'm back again. I'm running

pulsar-perf

produce command to publish 1 million messages, but it is pushing more than that. Not able to understand why it is happening, sometimes it pushes 1.7 million, or sometimes it will push more than 2 million messages. Any help is appreciated. Thank you!

We've just released Apache Pulsar Helm Chart 4.1.0 🎉 The official source release, as well as the binary Helm Chart release, are available at https://downloads.apache.org/pulsar/helm-chart/4.1.0/. The helm chart index at https://pulsar.apache.org/charts/ has been updated and the release is also available directly via helm. Release Notes: https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-4.1.0 Docs: https://github.com/apache/pulsar-helm-chart#readme and https://pulsar.apache.org/docs/helm-overview ArtifactHub: https://artifacthub.io/packages/helm/apache/pulsar/4.1.0 Thanks to all the contributors who made this possible.

Hi All, Am facing below issue on openshift : 2025-05-28T140721,461 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server java.io.IOException: Failed to load RocksDB JNI library at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:116) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:107) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:74) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:73) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:281) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:236) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.BookieResources.createLedgerStorage(BookieResources.java:107) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.EmbeddedServer$Builder.build(EmbeddedServer.java:410) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:277) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.doMain(Main.java:216) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.main(Main.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Caused by: java.lang.ExceptionInInitializerError at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared library at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:73) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more Caused by: java.io.IOException: Read-only file system at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?] at java.io.File.createTempFile(File.java:2170) ~[?:?] at java.io.File.createTempFile(File.java:2216) ~[?:?] at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:132) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:102) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:82) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:70) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more pulsar is 3.0.6 .any help here will be appreciated

According to this, it might be a JDK-related issue

We are using Pulsar 4.0.5 MongoDB IO connector and after configuring everything, the before document is not present in the message. After looking into more details, I have found that the latest IO connector is using Debzium 1.9.7 which does not support change.mode

change_streams_with_pre_image

config mentioned in the Debezium. Anyone else faced it and found a fix on this?

After upgraded to 3.0.11, i saw lot of logs like

org.apache.zookeeper.server.quorum.Learner - Revalidating client:0x3015fd17ce80066

and also on broker side some logs like it first closed the topic

2025-05-29T21:22:26,341+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [<persistent://6614/Salesforce/New> Contact] Topic closed

then it will try to create the topic almost immediately but it may fail for once then retry creating then succeed. any idea why it is doing this ? btw there is error in client side

Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: {"errorMsg":"org.apache.bookkeeper.mledger.ManagedLedgerException: org.apache.bookkeeper.mledger.ManagedLedgerException$BadVersionException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /managed-ledgers

Hello everyone, I need help understanding the behavior of shared subscriptions in a geo-replicated setup. We have several topics in our Pulsar deployment. For some topics (cache-related), we have geo-replication disabled. Others work as expected — the subscription cursor is replicated to the backup cluster. However, we are seeing inconsistent behavior with topics used for delayed messages and shared subscriptions. These topics have geo-replication enabled and use individual acknowledgments. According to the documentation, individual acknowledgments themselves are not replicated across clusters. However, the markDeletePosition should be replicated. In our tests, we noticed that the markDeletePosition in the backup cluster does not move predictably. In some cases, it remains unchanged for a long time. The only time it eventually advances is after the primary cluster stops receiving new messages to that topic — and then, after a delay, the markDeletePosition is finally updated in the backup cluster. Why does this happen, and is there a way to make markDeletePosition replication more predictable for topics with delayed messages and shared subscriptions? Pulsar version: 4.0.4 Thank you in advance!

Hello, question, my pulsar cluster added this configuration: allowAutoTopicCreation: "false" The idea is to prevent dynamic creations by the apps and manage all with pulsar-admin, but I'd like to know if it also affects the creation of the deadletter topics.

anybody knows this warn message

failed to get Partitioned metadata : {\"errorMsg\":\"org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /managed-ledgers/15548/Orchestration/persistent/closed_periods\",\"reqId\":2511449849216011802, \"remote\":\"pulsar-broker.pulsar.svc.cluster.local.analytics.svc.cluster.local/10.1.177.78:6650\", \"local\":\"/10.11.68.215:34530\"}","logger":"o.a.p.c.i.BinaryProtoLookupService","thread":"pulsar-client-io-5-1","level":"WARN","stacktrace":"<#f1bccd48> o.a.p.c.a.PulsarClientException$BrokerMetadataException: {\"errorMsg\":\"org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /managed-ledgers/15548/Orchestration/persistent/closed_periods\",\"reqId\":2511449849216011802, \"remote\":\"pulsar-broker.pulsar.svc.cluster.local.analytics.svc.cluster.local/10.1.177.78:6650\", \"local\":\"/10.11.68.215:34530\"}\n\tat o.a.p.c.i.ClientCnx.getPulsarClientException(ClientCnx.java:1352)\n\tat o.a.p.c.i.ClientCnx.handlePartitionResponse(ClientCnx.java:691)\n\tat o.a.p.c.p.PulsarDecoder.channelRead(PulsarDecoder.java:144)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.h.c.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat o.a.p.s.i.n.h.c.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.h.f.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.c.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat o.a.p.s.i.n.c.Abst...\n"}

btw subscription was created successfully after this . but it sometimes gave this warn saying connection loss. this only happened after 3.0.11 . did we change anything related ? it is just annoying

Is there anyway we can track the subscription creation time ?

Hello! question, I can't find in the pulsar documentation 4.0 the page of Pulsar SQL, is this feature available?

Hi everyone*,* I’m working on a real-time pub-sub system where data durability is not a requirement — I’m okay with potential data loss, and no message retention is needed. While exploring Apache Pulsar, I came across non-persistent topics, which seem ideal since they skip writing to BookKeeper. However, even with non-persistent topics, it appears that BookKeeper is still a required component of the Pulsar stack. For a setup where durability isn't needed at all, this seems like unnecessary overhead. Is there any way to eliminate the BookKeeper layer entirely in such a use case — perhaps running only the Pulsar broker (or a lighter version). I’d love to hear thoughts on the ideal architecture for this type of real-time, fire-and-forget messaging requirement. Thanks in advance!

Whops, just wanted to take a look at the slack integrations, didn’t mean to add anything 😂

Hi ! Using pusar-rs, we can see on a subscription a negative 'availablePermits' (-127), is it expected to have something < 0 ?

Hello, I can see there is activity working in this branch in flink pulsar connector: https://github.com/streamnative/flink-connector-pulsar/tree/v4.3-sn Will those changes be merged to the official filnk pulsar connector repository? The main repo looks like abandoned because there is no new releases since Dec 2023, and there are new PR merged in main that still weren't published 😞