Hi there, I'm running the command, but the perf tool is not consuming the messages. Can anyone help me with this? Adding screenshots if this helps

pulsar-perf consume --service-url  <pulsar://localhost:6650> --subscription-type Shared --rate 10000 --num-subscriptions 10 --num-consumers 10 --subscription-position Earliest

Update: If I run

pulsar-perf consume --service-url  <pulsar://localhost:6650>

<persistent://public/default/my-topic>

then, it runs fine! Most likely, one of the parameters is causing the issue.

pulsarlogo 📣 [ANNOUNCE] Apache Pulsar 3.0.12, 3.3.7 and 4.0.5 released 📣 pulsarlogo For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: 3.0.12: https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.12/ 3.3.7: https://pulsar.apache.org/release-notes/versioned/pulsar-3.3.7/ 4.0.5: https://pulsar.apache.org/release-notes/versioned/pulsar-4.0.5/ Although 3.3.7 was released, please notice that support for 3.3.x has already ended and no new releases are planned.

Hi there, 👋 I'm back again. I'm running

pulsar-perf

produce command to publish 1 million messages, but it is pushing more than that. Not able to understand why it is happening, sometimes it pushes 1.7 million, or sometimes it will push more than 2 million messages. Any help is appreciated. Thank you!

We've just released Apache Pulsar Helm Chart 4.1.0 🎉 The official source release, as well as the binary Helm Chart release, are available at https://downloads.apache.org/pulsar/helm-chart/4.1.0/. The helm chart index at https://pulsar.apache.org/charts/ has been updated and the release is also available directly via helm. Release Notes: https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-4.1.0 Docs: https://github.com/apache/pulsar-helm-chart#readme and https://pulsar.apache.org/docs/helm-overview ArtifactHub: https://artifacthub.io/packages/helm/apache/pulsar/4.1.0 Thanks to all the contributors who made this possible.

Hi All, Am facing below issue on openshift : 2025-05-28T140721,461 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server java.io.IOException: Failed to load RocksDB JNI library at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:116) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:107) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:74) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:73) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:281) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:236) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.bookie.BookieResources.createLedgerStorage(BookieResources.java:107) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.EmbeddedServer$Builder.build(EmbeddedServer.java:410) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:277) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.doMain(Main.java:216) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.server.Main.main(Main.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Caused by: java.lang.ExceptionInInitializerError at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared library at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:73) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more Caused by: java.io.IOException: Read-only file system at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?] at java.io.File.createTempFile(File.java:2170) ~[?:?] at java.io.File.createTempFile(File.java:2216) ~[?:?] at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:132) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:102) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:82) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:70) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] ... 11 more pulsar is 3.0.6 .any help here will be appreciated

According to this, it might be a JDK-related issue

We are using Pulsar 4.0.5 MongoDB IO connector and after configuring everything, the before document is not present in the message. After looking into more details, I have found that the latest IO connector is using Debzium 1.9.7 which does not support change.mode

change_streams_with_pre_image

config mentioned in the Debezium. Anyone else faced it and found a fix on this?

After upgraded to 3.0.11, i saw lot of logs like

org.apache.zookeeper.server.quorum.Learner - Revalidating client:0x3015fd17ce80066

and also on broker side some logs like it first closed the topic

2025-05-29T21:22:26,341+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [<persistent://6614/Salesforce/New> Contact] Topic closed

then it will try to create the topic almost immediately but it may fail for once then retry creating then succeed. any idea why it is doing this ? btw there is error in client side

Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: {"errorMsg":"org.apache.bookkeeper.mledger.ManagedLedgerException: org.apache.bookkeeper.mledger.ManagedLedgerException$BadVersionException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /managed-ledgers

Hello everyone, I need help understanding the behavior of shared subscriptions in a geo-replicated setup. We have several topics in our Pulsar deployment. For some topics (cache-related), we have geo-replication disabled. Others work as expected — the subscription cursor is replicated to the backup cluster. However, we are seeing inconsistent behavior with topics used for delayed messages and shared subscriptions. These topics have geo-replication enabled and use individual acknowledgments. According to the documentation, individual acknowledgments themselves are not replicated across clusters. However, the markDeletePosition should be replicated. In our tests, we noticed that the markDeletePosition in the backup cluster does not move predictably. In some cases, it remains unchanged for a long time. The only time it eventually advances is after the primary cluster stops receiving new messages to that topic — and then, after a delay, the markDeletePosition is finally updated in the backup cluster. Why does this happen, and is there a way to make markDeletePosition replication more predictable for topics with delayed messages and shared subscriptions? Pulsar version: 4.0.4 Thank you in advance!

Hello, question, my pulsar cluster added this configuration: allowAutoTopicCreation: "false" The idea is to prevent dynamic creations by the apps and manage all with pulsar-admin, but I'd like to know if it also affects the creation of the deadletter topics.

anybody knows this warn message

failed to get Partitioned metadata : {\"errorMsg\":\"org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /managed-ledgers/15548/Orchestration/persistent/closed_periods\",\"reqId\":2511449849216011802, \"remote\":\"pulsar-broker.pulsar.svc.cluster.local.analytics.svc.cluster.local/10.1.177.78:6650\", \"local\":\"/10.11.68.215:34530\"}","logger":"o.a.p.c.i.BinaryProtoLookupService","thread":"pulsar-client-io-5-1","level":"WARN","stacktrace":"<#f1bccd48> o.a.p.c.a.PulsarClientException$BrokerMetadataException: {\"errorMsg\":\"org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /managed-ledgers/15548/Orchestration/persistent/closed_periods\",\"reqId\":2511449849216011802, \"remote\":\"pulsar-broker.pulsar.svc.cluster.local.analytics.svc.cluster.local/10.1.177.78:6650\", \"local\":\"/10.11.68.215:34530\"}\n\tat o.a.p.c.i.ClientCnx.getPulsarClientException(ClientCnx.java:1352)\n\tat o.a.p.c.i.ClientCnx.handlePartitionResponse(ClientCnx.java:691)\n\tat o.a.p.c.p.PulsarDecoder.channelRead(PulsarDecoder.java:144)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.h.c.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat o.a.p.s.i.n.h.c.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.h.f.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat o.a.p.s.i.n.c.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)\n\tat o.a.p.s.i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat o.a.p.s.i.n.c.Abst...\n"}

btw subscription was created successfully after this . but it sometimes gave this warn saying connection loss. this only happened after 3.0.11 . did we change anything related ? it is just annoying

Is there anyway we can track the subscription creation time ?

Hello! question, I can't find in the pulsar documentation 4.0 the page of Pulsar SQL, is this feature available?

Hi everyone*,* I’m working on a real-time pub-sub system where data durability is not a requirement — I’m okay with potential data loss, and no message retention is needed. While exploring Apache Pulsar, I came across non-persistent topics, which seem ideal since they skip writing to BookKeeper. However, even with non-persistent topics, it appears that BookKeeper is still a required component of the Pulsar stack. For a setup where durability isn't needed at all, this seems like unnecessary overhead. Is there any way to eliminate the BookKeeper layer entirely in such a use case — perhaps running only the Pulsar broker (or a lighter version). I’d love to hear thoughts on the ideal architecture for this type of real-time, fire-and-forget messaging requirement. Thanks in advance!

Whops, just wanted to take a look at the slack integrations, didn’t mean to add anything 😂

Hi ! Using pusar-rs, we can see on a subscription a negative 'availablePermits' (-127), is it expected to have something < 0 ?

Hello, I can see there is activity working in this branch in flink pulsar connector: https://github.com/streamnative/flink-connector-pulsar/tree/v4.3-sn Will those changes be merged to the official filnk pulsar connector repository? The main repo looks like abandoned because there is no new releases since Dec 2023, and there are new PR merged in main that still weren't published 😞

Hi everyone, wanted to know does non persistent partitioning of a topic supports broadcasting to all consumers for that particular topic?

Hello Team I have query on Node JS pulsar-client module here I am trying to serialize and deserialize messageId for my business use case but issue is for proper error handling if in case we pass invalid buffer data to MessageId.deserialize method then it throws C++ exception which is not handled by pulsar-client and node JS crashes did anyone else face this issue? I see similar issue was there while creating a client too but that got fixed with this issue and PR https://github.com/apache/pulsar-client-node/pull/307/files where it was wrapped with try catch block but deserialize is still not fixed any C++ experts can help to contribute with the fix or a solution would be appreciated

hello everyone I need a tool for monitoring apache pulsar 4.0.0 version but I got a customAuthentication for my architecture. Can some one help me ?

I'm fairly new to pulsar and really just evaluating it as a replacement for kafka. I have some questions about data offloading to s3. How does this effect replication with in pulsar? is it really necessary to do both? if we were to lower the offloading time / size to be fairly low could we lower replication factor? One of out biggest costs with kafka right now is data transfer across availability zones in AWS. either from broker <> broker or broker <> clients in different zones. if our data lives on s3 the need for large disk is reduce and may reduce the need for replication rather significantly if I'm understanding offloading correctly. we'd basically only need enough disk to account for data that hasn't been moved yet. Is this accurate? is it an appropriate use of offloading?

If we set our offloading threshold to something like 10 minutes. Is that silly?

Hey, We're hoping you can provide some insight into an issue one of our clients experienced following a recent rolling upgrade of our Pulsar cluster. Context: • Cluster Version: Pulsar 3.0.11 (High-Availability Setup) • Action: We performed a rolling upgrade of all components in production. • Client: One of our customers runs multiple Pulsar Functions. The Issue: Immediately after the upgrade, two of the customer's functions became unresponsive and started logging the following exception:

java.lang.IllegalStateException: Field 'message' is not set

at org.apache.pulsar.common.api.proto.CommandLookupTopicResponse.getMessage(CommandLookupTopicResponse.java:220)

at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:629)

at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:154)

The issue persisted until the function pods were manually restarted, by which time their own customers had already been impacted. • ClientCnx.handleLookupResponse --> ClientCnx represents the client's connection to a broker. It was processing the response to a CommandLookupTopic request. A topic lookup is the mechanism a client uses to ask the Pulsar cluster, "Which broker is currently serving this topic?". • CommandLookupTopicResponse.getMessage --> The client received a response from the broker. The code then attempted to get an error message from this response. • IllegalStateException: Field 'message' is not set --> This exception means the client code expected the message field to be populated in the response it received, but it wasn't. In the Pulsar protocol, this message field is optional. The broker sent a response indicating failure or redirection but didn't include the optional descriptive text. Our Hypothesis: Our theory is that the function's internal client entered a "stuck" state during the broker rolling upgrade and failed to recover. What happened during the update: 1. A broker pod is terminated 2. All client connections to that broker are severed. 3. The Pulsar Function's internal client automatically tries to reconnect. 4. As part of the reconnection, it performs a topic lookup. 5. The ownership of the topic may be in the process of being transferred from the old broker to a new one. If the client's lookup request hits a broker during this transient state, the broker might issue a Redirect or Failed lookup response. 6. In some edge cases, this response is sent without the optional message field, triggering the bug in the We also discovered a version mismatch between the cluster and the client libraries: • *Client Libraries:*pulsar-client and pulsar-functions-api are both version 3.2.1. • Cluster Version:3.0.11. My question are now: Is it a requirement for the client library versions to strictly match the Pulsar cluster version? Could this version skew be a potential cause for this kind of failure during a rolling upgrade? Are you aware of any existing GitHub issues related to this specific IllegalStateException in this context? We were unable to find a direct match in our own research. Thanks in advance!

I am trying to setup a pulsar cluster with only ETCD as the metadata store. But now I am getting stuck with using the ETCD as the metadata store for the bookies, it seems like it cannot recognize ETCD. can someone help me about the setting?

Hello, I run Pulsar in cluster mode (3.0.8), with 3 zookeepers / 3 bookkeepers / 5 brokers / 5 proxy. I had a server restart and it was hosting : 1 broker, 1 proxy and 1 zookeeper (The broker and the zookeeper instances were the leaders). Even when my server came back up, almost all the topics balanced themselves on only two of the brokers (even brokers that stayed up lost their topics) as you can see on the attached capture : Now, the existing (and newly created) topics are only managed by only 2 brokers (out of 5) and not balanced between all the others. Can you help me to find a way to prevent this behaviour please ? How can I achieve effective load balancing when a situation like this occurs. Is it possible to force the balancing of topics between brokers without restarting them ? For availability and Production reasons, I want to avoid downtime as most as possible. we got this configuration for broker.conf (loadbalancer conf):

loadBalancerEnabled=true
loadBalancerReportUpdateThresholdPercentage=10
loadBalancerReportUpdateMinIntervalMillis=5000
loadBalancerReportUpdateMaxIntervalMinutes=15
loadBalancerHostUsageCheckIntervalMinutes=1
loadBalancerSheddingEnabled=true
loadBalancerSheddingIntervalMinutes=1
loadBalancerSheddingGracePeriodMinutes=30
loadBalancerBrokerMaxTopics=50000
loadBalancerBrokerOverloadedThresholdPercentage=85
loadBalancerResourceQuotaUpdateIntervalMinutes=15
loadBalancerAutoBundleSplitEnabled=true
loadBalancerAutoUnloadSplitBundlesEnabled=true
loadBalancerNamespaceBundleMaxTopics=1000
loadBalancerNamespaceBundleMaxSessions=1000
loadBalancerNamespaceBundleMaxMsgRate=30000
loadBalancerNamespaceBundleMaxBandwidthMbytes=100
loadBalancerNamespaceMaximumBundles=128
loadBalancerOverrideBrokerNicSpeedGbps=
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.LeastLongTermMessageRate
loadBalancerBrokerThresholdShedderPercentage=10
loadBalancerAverageResourceUsageDifferenceThresholdPercentage=10
loadBalancerMsgRateDifferenceShedderThreshold=50
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold=4
loadBalancerHistoryResourcePercentage=0.9
loadBalancerBandwithInResourceWeight=1.0
loadBalancerBandwithOutResourceWeight=1.0
loadBalancerCPUResourceWeight=1.0
loadBalancerDirectMemoryResourceWeight=1.0
loadBalancerBundleUnloadMinThroughputThreshold=10
loadBalancerAvgShedderLowThreshold = 15
loadBalancerAvgShedderHighThreshold = 40
loadBalancerAvgShedderHitCountLowThreshold = 8
loadBalancerAvgShedderHitCountHighThreshold = 2
loadBalancerDebugModeEnabled=false
loadBalancerBrokerLoadTargetStd=0.25
loadBalancerSheddingConditionHitCountThreshold=3
loadBalancerTransferEnabled=true
loadBalancerMaxNumberOfBrokerSheddingPerCycle=3
loadBalancerBrokerLoadDataTTLInSeconds=1800
loadBalancerMaxNumberOfBundlesInBundleLoadReport=10
loadBalancerSplitIntervalMinutes=1
loadBalancerMaxNumberOfBundlesToSplitPerCycle=10
loadBalancerNamespaceBundleSplitConditionHitCountThreshold=3
loadBalancerServiceUnitStateTombstoneDelayTimeInSeconds=3600
loadBalancerMemoryResourceWeight=1.0

Hello! I am doing some geo-replication testing and I have followed the documentation to connect and test my two pulsar clusters. I have gotten it to work, but I have to manually create my topics on the cluster for it to work, and the topic is not being replicated to the other cluster, so I have to manually do a topics create on both clusters. Per documentation I should be able to configure it: • Once you create a geo-replication namespace, any topics that producers or consumers create within that namespace are replicated across clusters. Typically, each application uses the

serviceUrl

for the local cluster. I enabled geo-replication at the namespace level, not at the topic level. Does anyone know how to solve this issue? Using v4.0.1

Hey, I have a question about using Pulsar as a message queue for our use case. Our scenario: • We use a topic as a queue that receives bursts of messages (small amounts) once or twice a day • Processing time is very long (15-80 minutes per message) and resource-intensive • We monitor the topic backlog and auto-scale consumers accordingly The problem: We can't get messages to distribute evenly across consumers. Ideally, we want 1 message per consumer, but what happens is the first consumer that gets up claims ALL messages from the backlog. When we scale up and add more consumers, they sit idle because all messages are already assigned to the first consumer. This creates a situation where 1 consumer processes for hours while 9 others do nothing, even though we have 10 messages and 10 consumers. I've opened a GitHub discussion with more details: https://github.com/apache/pulsar/discussions/24477 Question: Is there a broker-side configuration to limit how many messages each consumer can claim? Or is there a different pattern we should use for long-processing tasks with dynamic scaling? Any insights would be really appreciated!

Does anyone deployed Pulsar on Kubernetes using local PVC on production? Our current cloud provider has relatively slow Ceph-based attached volumes, and fast local NVMe SSD. We already to have replication on bookkeeper, so probably we don't need to add another layer like Longhorn. I found that there is such an option in Helm, but the question is - is it supported?