Hi team! We noted an issue in our production cluster. I found an earlier discussion that is relevant and I have added the details in the comment -- https://github.com/apache/pulsar/issues/5202#issuecomment-2882929294 Can someone take a look at this? Crux: The backlog reported for the topic was 30K+ whereas there was only 1 ledger. I see "Message expiry failed - mark delete failed" in broker logs during this period and the issue resolved on its own.

Hi team, I have a quick question, docs says recommended version of Java for pulsar > 2.10 is 17 here https://github.com/apache/pulsar/blob/master/README.md#pulsar-runtime-java-version-recommendation I'm currently using pulsar 3.2.3, will there be any compatibility issues if I use Java 21 as runtime? Should I have to upgrade my pulsar version to 4.0.4 which is latest? I have tried locally running pulsar 3.2.3 with java 21. I was able to start the standalone cluster and create topics etc

Hi 👋 1. I'm using the Go client

v0.15.0

and I wanted to ask if there a way to get messages from compacted topics other than using the

TableView

? I'm looking for a way to read messages from a stream (use of a channel) rather than storing all them at once in a

map[string]any

like the

TableView

does. It's for at least half a million records after compaction. 2. So I tried using a

Reader

with

ReadCompacted

option set to

true

but I'm getting all the messages: is there a way to actually read from the compacted topic with a

Reader

? (I'm using persistent topics) Thank you

Hi there 👋 I have set up Apache Pulsar and pushed 100 messages. While creating the namespace, I set subscriptionExpirationTime to 1 minute. On the consumer side, I've defined ConsumerType to be shared. After acknowledging all the messages, there are no subscribers, as they get removed based on my config. However, I'm wondering why space/storage is not freed up. As the docs say, once all the subscribers acknowledge the messages, it will be removed from the Pulsar system, so space should be freed. PS: Retention is already disabled. Can someone please help me understand how the messages will be removed from the broker and how space will be freed? Any help is appreciated.

hello everyone can someone help me about apachepulsar ? I got 8 server 3 zookeper 5 bookie and brokers and suddenly bookies die with no error ( I cant catch the error )

Pulsar site search on pulsar.apache.org is working again. It wasn't working for several months due to ASF Privacy policy changes (more details in GH discussion).

Hi there 👋 , I'm trying to create a topic using pulsar_perf and getting the following error. Can someone please help me fix it?

Topic creation encountered an exception by initialization topic policies service. org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException cluster-a

Hi there, I'm running the command, but the perf tool is not consuming the messages. Can anyone help me with this? Adding screenshots if this helps

pulsar-perf consume --service-url  <pulsar://localhost:6650> --subscription-type Shared --rate 10000 --num-subscriptions 10 --num-consumers 10 --subscription-position Earliest

Update: If I run

pulsar-perf consume --service-url  <pulsar://localhost:6650>

<persistent://public/default/my-topic>

then, it runs fine! Most likely, one of the parameters is causing the issue.

pulsarlogo 📣 [ANNOUNCE] Apache Pulsar 3.0.12, 3.3.7 and 4.0.5 released 📣 pulsarlogo For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: 3.0.12: https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.12/ 3.3.7: https://pulsar.apache.org/release-notes/versioned/pulsar-3.3.7/ 4.0.5: https://pulsar.apache.org/release-notes/versioned/pulsar-4.0.5/ Although 3.3.7 was released, please notice that support for 3.3.x has already ended and no new releases are planned.

Hi there, 👋 I'm back again. I'm running

pulsar-perf

produce command to publish 1 million messages, but it is pushing more than that. Not able to understand why it is happening, sometimes it pushes 1.7 million, or sometimes it will push more than 2 million messages. Any help is appreciated. Thank you!

We've just released Apache Pulsar Helm Chart 4.1.0 🎉 The official source release, as well as the binary Helm Chart release, are available at https://downloads.apache.org/pulsar/helm-chart/4.1.0/. The helm chart index at https://pulsar.apache.org/charts/ has been updated and the release is also available directly via helm. Release Notes: https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-4.1.0 Docs: https://github.com/apache/pulsar-helm-chart#readme and https://pulsar.apache.org/docs/helm-overview ArtifactHub: https://artifacthub.io/packages/helm/apache/pulsar/4.1.0 Thanks to all the contributors who made this possible.

Hi All, Am facing below issue on openshift : 2025-05-28T140721,461 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server java.io.IOException: Failed to load RocksDB JNI library at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:116) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:107) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:74) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:73) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:281) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:236) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.BookieResources.createLedgerStorage(BookieResources.java:107) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.EmbeddedServer$Builder.build(EmbeddedServer.java:410) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:277) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.doMain(Main.java:216) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.main(Main.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Caused by: java.lang.ExceptionInInitializerError at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared library at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:73) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more Caused by: java.io.IOException: Read-only file system at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?] at java.io.File.createTempFile(File.java:2170) ~[?:?] at java.io.File.createTempFile(File.java:2216) ~[?:?] at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:132) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:102) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:82) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:70) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more pulsar is 3.0.6 .any help here will be appreciated

According to this, it might be a JDK-related issue

We are using Pulsar 4.0.5 MongoDB IO connector and after configuring everything, the before document is not present in the message. After looking into more details, I have found that the latest IO connector is using Debzium 1.9.7 which does not support change.mode

change_streams_with_pre_image

config mentioned in the Debezium. Anyone else faced it and found a fix on this?

After upgraded to 3.0.11, i saw lot of logs like

org.apache.zookeeper.server.quorum.Learner - Revalidating client:0x3015fd17ce80066

and also on broker side some logs like it first closed the topic

2025-05-29T21:22:26,341+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [<persistent://6614/Salesforce/New> Contact] Topic closed

then it will try to create the topic almost immediately but it may fail for once then retry creating then succeed. any idea why it is doing this ? btw there is error in client side

Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: {"errorMsg":"org.apache.bookkeeper.mledger.ManagedLedgerException: org.apache.bookkeeper.mledger.ManagedLedgerException$BadVersionException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /managed-ledgers

Hello everyone, I need help understanding the behavior of shared subscriptions in a geo-replicated setup. We have several topics in our Pulsar deployment. For some topics (cache-related), we have geo-replication disabled. Others work as expected — the subscription cursor is replicated to the backup cluster. However, we are seeing inconsistent behavior with topics used for delayed messages and shared subscriptions. These topics have geo-replication enabled and use individual acknowledgments. According to the documentation, individual acknowledgments themselves are not replicated across clusters. However, the markDeletePosition should be replicated. In our tests, we noticed that the markDeletePosition in the backup cluster does not move predictably. In some cases, it remains unchanged for a long time. The only time it eventually advances is after the primary cluster stops receiving new messages to that topic — and then, after a delay, the markDeletePosition is finally updated in the backup cluster. Why does this happen, and is there a way to make markDeletePosition replication more predictable for topics with delayed messages and shared subscriptions? Pulsar version: 4.0.4 Thank you in advance!

Hello, question, my pulsar cluster added this configuration: allowAutoTopicCreation: "false" The idea is to prevent dynamic creations by the apps and manage all with pulsar-admin, but I'd like to know if it also affects the creation of the deadletter topics.

anybody knows this warn message

failed to get Partitioned metadata : {\"errorMsg\":\"org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /managed-ledgers/15548/Orchestration/persistent/closed_periods\",\"reqId\":2511449849216011802, \"remote\":\"pulsar-broker.pulsar.svc.cluster.local.analytics.svc.cluster.local/10.1.177.78:6650\", \"local\":\"/10.11.68.215:34530\"}","logger":"o.a.p.c.i.BinaryProtoLookupService","thread":"pulsar-client-io-5-1","level":"WARN","stacktrace":"<#f1bccd48> o.a.p.c.a.PulsarClientException$BrokerMetadataException: {\"errorMsg\":\"org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /managed-ledgers/15548/Orchestration/persistent/closed_periods\",\"reqId\":2511449849216011802, \"remote\":\"pulsar-broker.pulsar.svc.cluster.local.analytics.svc.cluster.local/10.1.177.78:6650\", \"local\":\"/10.11.68.215:34530\"}\n\tat o.a.p.c.i.ClientCnx.getPulsarClientException(ClientCnx.java:1352)\n\tat o.a.p.c.i.ClientCnx.handlePartitionResponse(ClientCnx.java:691)\n\tat o.a.p.c.p.PulsarDecoder.channelRead(PulsarDecoder.java:144)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.h.c.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat o.a.p.s.i.n.h.c.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.h.f.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.c.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat o.a.p.s.i.n.c.Abst...\n"}

btw subscription was created successfully after this . but it sometimes gave this warn saying connection loss. this only happened after 3.0.11 . did we change anything related ? it is just annoying

Is there anyway we can track the subscription creation time ?

Hello! question, I can't find in the pulsar documentation 4.0 the page of Pulsar SQL, is this feature available?

Hi everyone*,* I’m working on a real-time pub-sub system where data durability is not a requirement — I’m okay with potential data loss, and no message retention is needed. While exploring Apache Pulsar, I came across non-persistent topics, which seem ideal since they skip writing to BookKeeper. However, even with non-persistent topics, it appears that BookKeeper is still a required component of the Pulsar stack. For a setup where durability isn't needed at all, this seems like unnecessary overhead. Is there any way to eliminate the BookKeeper layer entirely in such a use case — perhaps running only the Pulsar broker (or a lighter version). I’d love to hear thoughts on the ideal architecture for this type of real-time, fire-and-forget messaging requirement. Thanks in advance!

Whops, just wanted to take a look at the slack integrations, didn’t mean to add anything 😂

Hi ! Using pusar-rs, we can see on a subscription a negative 'availablePermits' (-127), is it expected to have something < 0 ?

Hello, I can see there is activity working in this branch in flink pulsar connector: https://github.com/streamnative/flink-connector-pulsar/tree/v4.3-sn Will those changes be merged to the official filnk pulsar connector repository? The main repo looks like abandoned because there is no new releases since Dec 2023, and there are new PR merged in main that still weren't published 😞

Hi everyone, wanted to know does non persistent partitioning of a topic supports broadcasting to all consumers for that particular topic?

Hello Team I have query on Node JS pulsar-client module here I am trying to serialize and deserialize messageId for my business use case but issue is for proper error handling if in case we pass invalid buffer data to MessageId.deserialize method then it throws C++ exception which is not handled by pulsar-client and node JS crashes did anyone else face this issue? I see similar issue was there while creating a client too but that got fixed with this issue and PR https://github.com/apache/pulsar-client-node/pull/307/files where it was wrapped with try catch block but deserialize is still not fixed any C++ experts can help to contribute with the fix or a solution would be appreciated

hello everyone I need a tool for monitoring apache pulsar 4.0.0 version but I got a customAuthentication for my architecture. Can some one help me ?

I'm fairly new to pulsar and really just evaluating it as a replacement for kafka. I have some questions about data offloading to s3. How does this effect replication with in pulsar? is it really necessary to do both? if we were to lower the offloading time / size to be fairly low could we lower replication factor? One of out biggest costs with kafka right now is data transfer across availability zones in AWS. either from broker <> broker or broker <> clients in different zones. if our data lives on s3 the need for large disk is reduce and may reduce the need for replication rather significantly if I'm understanding offloading correctly. we'd basically only need enough disk to account for data that hasn't been moved yet. Is this accurate? is it an appropriate use of offloading?

If we set our offloading threshold to something like 10 minutes. Is that silly?

Hey, We're hoping you can provide some insight into an issue one of our clients experienced following a recent rolling upgrade of our Pulsar cluster. Context: • Cluster Version: Pulsar 3.0.11 (High-Availability Setup) • Action: We performed a rolling upgrade of all components in production. • Client: One of our customers runs multiple Pulsar Functions. The Issue: Immediately after the upgrade, two of the customer's functions became unresponsive and started logging the following exception:

java.lang.IllegalStateException: Field 'message' is not set

at org.apache.pulsar.common.api.proto.CommandLookupTopicResponse.getMessage(CommandLookupTopicResponse.java:220)

at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:629)

at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:154)

The issue persisted until the function pods were manually restarted, by which time their own customers had already been impacted. • ClientCnx.handleLookupResponse --> ClientCnx represents the client's connection to a broker. It was processing the response to a CommandLookupTopic request. A topic lookup is the mechanism a client uses to ask the Pulsar cluster, "Which broker is currently serving this topic?". • CommandLookupTopicResponse.getMessage --> The client received a response from the broker. The code then attempted to get an error message from this response. • IllegalStateException: Field 'message' is not set --> This exception means the client code expected the message field to be populated in the response it received, but it wasn't. In the Pulsar protocol, this message field is optional. The broker sent a response indicating failure or redirection but didn't include the optional descriptive text. Our Hypothesis: Our theory is that the function's internal client entered a "stuck" state during the broker rolling upgrade and failed to recover. What happened during the update: 1. A broker pod is terminated 2. All client connections to that broker are severed. 3. The Pulsar Function's internal client automatically tries to reconnect. 4. As part of the reconnection, it performs a topic lookup. 5. The ownership of the topic may be in the process of being transferred from the old broker to a new one. If the client's lookup request hits a broker during this transient state, the broker might issue a Redirect or Failed lookup response. 6. In some edge cases, this response is sent without the optional message field, triggering the bug in the We also discovered a version mismatch between the cluster and the client libraries: • *Client Libraries:*pulsar-client and pulsar-functions-api are both version 3.2.1. • Cluster Version:3.0.11. My question are now: Is it a requirement for the client library versions to strictly match the Pulsar cluster version? Could this version skew be a potential cause for this kind of failure during a rolling upgrade? Are you aware of any existing GitHub issues related to this specific IllegalStateException in this context? We were unable to find a direct match in our own research. Thanks in advance!