Hello! question, I can't find in the pulsar documentation 4.0 the page of Pulsar SQL, is this feature available?

Hi everyone*,* I’m working on a real-time pub-sub system where data durability is not a requirement — I’m okay with potential data loss, and no message retention is needed. While exploring Apache Pulsar, I came across non-persistent topics, which seem ideal since they skip writing to BookKeeper. However, even with non-persistent topics, it appears that BookKeeper is still a required component of the Pulsar stack. For a setup where durability isn't needed at all, this seems like unnecessary overhead. Is there any way to eliminate the BookKeeper layer entirely in such a use case — perhaps running only the Pulsar broker (or a lighter version). I’d love to hear thoughts on the ideal architecture for this type of real-time, fire-and-forget messaging requirement. Thanks in advance!

Whops, just wanted to take a look at the slack integrations, didn’t mean to add anything 😂

Hi ! Using pusar-rs, we can see on a subscription a negative 'availablePermits' (-127), is it expected to have something < 0 ?

Hello, I can see there is activity working in this branch in flink pulsar connector: https://github.com/streamnative/flink-connector-pulsar/tree/v4.3-sn Will those changes be merged to the official filnk pulsar connector repository? The main repo looks like abandoned because there is no new releases since Dec 2023, and there are new PR merged in main that still weren't published 😞

Hi everyone, wanted to know does non persistent partitioning of a topic supports broadcasting to all consumers for that particular topic?

Hello Team I have query on Node JS pulsar-client module here I am trying to serialize and deserialize messageId for my business use case but issue is for proper error handling if in case we pass invalid buffer data to MessageId.deserialize method then it throws C++ exception which is not handled by pulsar-client and node JS crashes did anyone else face this issue? I see similar issue was there while creating a client too but that got fixed with this issue and PR https://github.com/apache/pulsar-client-node/pull/307/files where it was wrapped with try catch block but deserialize is still not fixed any C++ experts can help to contribute with the fix or a solution would be appreciated

hello everyone I need a tool for monitoring apache pulsar 4.0.0 version but I got a customAuthentication for my architecture. Can some one help me ?

I'm fairly new to pulsar and really just evaluating it as a replacement for kafka. I have some questions about data offloading to s3. How does this effect replication with in pulsar? is it really necessary to do both? if we were to lower the offloading time / size to be fairly low could we lower replication factor? One of out biggest costs with kafka right now is data transfer across availability zones in AWS. either from broker <> broker or broker <> clients in different zones. if our data lives on s3 the need for large disk is reduce and may reduce the need for replication rather significantly if I'm understanding offloading correctly. we'd basically only need enough disk to account for data that hasn't been moved yet. Is this accurate? is it an appropriate use of offloading?

If we set our offloading threshold to something like 10 minutes. Is that silly?

Hey, We're hoping you can provide some insight into an issue one of our clients experienced following a recent rolling upgrade of our Pulsar cluster. Context: • Cluster Version: Pulsar 3.0.11 (High-Availability Setup) • Action: We performed a rolling upgrade of all components in production. • Client: One of our customers runs multiple Pulsar Functions. The Issue: Immediately after the upgrade, two of the customer's functions became unresponsive and started logging the following exception:

java.lang.IllegalStateException: Field 'message' is not set

at org.apache.pulsar.common.api.proto.CommandLookupTopicResponse.getMessage(CommandLookupTopicResponse.java:220)

at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:629)

at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:154)

The issue persisted until the function pods were manually restarted, by which time their own customers had already been impacted. • ClientCnx.handleLookupResponse --> ClientCnx represents the client's connection to a broker. It was processing the response to a CommandLookupTopic request. A topic lookup is the mechanism a client uses to ask the Pulsar cluster, "Which broker is currently serving this topic?". • CommandLookupTopicResponse.getMessage --> The client received a response from the broker. The code then attempted to get an error message from this response. • IllegalStateException: Field 'message' is not set --> This exception means the client code expected the message field to be populated in the response it received, but it wasn't. In the Pulsar protocol, this message field is optional. The broker sent a response indicating failure or redirection but didn't include the optional descriptive text. Our Hypothesis: Our theory is that the function's internal client entered a "stuck" state during the broker rolling upgrade and failed to recover. What happened during the update: 1. A broker pod is terminated 2. All client connections to that broker are severed. 3. The Pulsar Function's internal client automatically tries to reconnect. 4. As part of the reconnection, it performs a topic lookup. 5. The ownership of the topic may be in the process of being transferred from the old broker to a new one. If the client's lookup request hits a broker during this transient state, the broker might issue a Redirect or Failed lookup response. 6. In some edge cases, this response is sent without the optional message field, triggering the bug in the We also discovered a version mismatch between the cluster and the client libraries: • *Client Libraries:*pulsar-client and pulsar-functions-api are both version 3.2.1. • Cluster Version:3.0.11. My question are now: Is it a requirement for the client library versions to strictly match the Pulsar cluster version? Could this version skew be a potential cause for this kind of failure during a rolling upgrade? Are you aware of any existing GitHub issues related to this specific IllegalStateException in this context? We were unable to find a direct match in our own research. Thanks in advance!

I am trying to setup a pulsar cluster with only ETCD as the metadata store. But now I am getting stuck with using the ETCD as the metadata store for the bookies, it seems like it cannot recognize ETCD. can someone help me about the setting?

Hello, I run Pulsar in cluster mode (3.0.8), with 3 zookeepers / 3 bookkeepers / 5 brokers / 5 proxy. I had a server restart and it was hosting : 1 broker, 1 proxy and 1 zookeeper (The broker and the zookeeper instances were the leaders). Even when my server came back up, almost all the topics balanced themselves on only two of the brokers (even brokers that stayed up lost their topics) as you can see on the attached capture : Now, the existing (and newly created) topics are only managed by only 2 brokers (out of 5) and not balanced between all the others. Can you help me to find a way to prevent this behaviour please ? How can I achieve effective load balancing when a situation like this occurs. Is it possible to force the balancing of topics between brokers without restarting them ? For availability and Production reasons, I want to avoid downtime as most as possible. we got this configuration for broker.conf (loadbalancer conf):

loadBalancerEnabled=true
loadBalancerReportUpdateThresholdPercentage=10
loadBalancerReportUpdateMinIntervalMillis=5000
loadBalancerReportUpdateMaxIntervalMinutes=15
loadBalancerHostUsageCheckIntervalMinutes=1
loadBalancerSheddingEnabled=true
loadBalancerSheddingIntervalMinutes=1
loadBalancerSheddingGracePeriodMinutes=30
loadBalancerBrokerMaxTopics=50000
loadBalancerBrokerOverloadedThresholdPercentage=85
loadBalancerResourceQuotaUpdateIntervalMinutes=15
loadBalancerAutoBundleSplitEnabled=true
loadBalancerAutoUnloadSplitBundlesEnabled=true
loadBalancerNamespaceBundleMaxTopics=1000
loadBalancerNamespaceBundleMaxSessions=1000
loadBalancerNamespaceBundleMaxMsgRate=30000
loadBalancerNamespaceBundleMaxBandwidthMbytes=100
loadBalancerNamespaceMaximumBundles=128
loadBalancerOverrideBrokerNicSpeedGbps=
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.LeastLongTermMessageRate
loadBalancerBrokerThresholdShedderPercentage=10
loadBalancerAverageResourceUsageDifferenceThresholdPercentage=10
loadBalancerMsgRateDifferenceShedderThreshold=50
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold=4
loadBalancerHistoryResourcePercentage=0.9
loadBalancerBandwithInResourceWeight=1.0
loadBalancerBandwithOutResourceWeight=1.0
loadBalancerCPUResourceWeight=1.0
loadBalancerDirectMemoryResourceWeight=1.0
loadBalancerBundleUnloadMinThroughputThreshold=10
loadBalancerAvgShedderLowThreshold = 15
loadBalancerAvgShedderHighThreshold = 40
loadBalancerAvgShedderHitCountLowThreshold = 8
loadBalancerAvgShedderHitCountHighThreshold = 2
loadBalancerDebugModeEnabled=false
loadBalancerBrokerLoadTargetStd=0.25
loadBalancerSheddingConditionHitCountThreshold=3
loadBalancerTransferEnabled=true
loadBalancerMaxNumberOfBrokerSheddingPerCycle=3
loadBalancerBrokerLoadDataTTLInSeconds=1800
loadBalancerMaxNumberOfBundlesInBundleLoadReport=10
loadBalancerSplitIntervalMinutes=1
loadBalancerMaxNumberOfBundlesToSplitPerCycle=10
loadBalancerNamespaceBundleSplitConditionHitCountThreshold=3
loadBalancerServiceUnitStateTombstoneDelayTimeInSeconds=3600
loadBalancerMemoryResourceWeight=1.0

Hello! I am doing some geo-replication testing and I have followed the documentation to connect and test my two pulsar clusters. I have gotten it to work, but I have to manually create my topics on the cluster for it to work, and the topic is not being replicated to the other cluster, so I have to manually do a topics create on both clusters. Per documentation I should be able to configure it: • Once you create a geo-replication namespace, any topics that producers or consumers create within that namespace are replicated across clusters. Typically, each application uses the

serviceUrl

for the local cluster. I enabled geo-replication at the namespace level, not at the topic level. Does anyone know how to solve this issue? Using v4.0.1

Hey, I have a question about using Pulsar as a message queue for our use case. Our scenario: • We use a topic as a queue that receives bursts of messages (small amounts) once or twice a day • Processing time is very long (15-80 minutes per message) and resource-intensive • We monitor the topic backlog and auto-scale consumers accordingly The problem: We can't get messages to distribute evenly across consumers. Ideally, we want 1 message per consumer, but what happens is the first consumer that gets up claims ALL messages from the backlog. When we scale up and add more consumers, they sit idle because all messages are already assigned to the first consumer. This creates a situation where 1 consumer processes for hours while 9 others do nothing, even though we have 10 messages and 10 consumers. I've opened a GitHub discussion with more details: https://github.com/apache/pulsar/discussions/24477 Question: Is there a broker-side configuration to limit how many messages each consumer can claim? Or is there a different pattern we should use for long-processing tasks with dynamic scaling? Any insights would be really appreciated!

Does anyone deployed Pulsar on Kubernetes using local PVC on production? Our current cloud provider has relatively slow Ceph-based attached volumes, and fast local NVMe SSD. We already to have replication on bookkeeper, so probably we don't need to add another layer like Longhorn. I found that there is such an option in Helm, but the question is - is it supported?

Hi All, In one of our deployment we have 3 partitioned topics and we have java producer code producing data to that topics . When i check the producer stats its showing like below where for same number of messages the latency is high and because of which i think the pending messages are also increasing . 2025-07-04 124036.832 [pulsar-timer-6-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://public/default/-partition-1] [-175-224582] Pending messages: 15 --- Publish throughput: 66.55 msg/s --- 0.49 Mbit/s --- Latency: med: *3000.*000 ms - 95pct: 4741.000 ms - 99pct: 5997.000 ms - 99.9pct: 6000.000 ms - max: 6404.000 ms --- BatchSize: med: 2.000 - 95pct: 65.000 - 99pct: 68.000 - 99.9pct: 69.000 - max: 69.000 --- MsgSize: med: 1930.000 bytes - 95pct: 64713.000 bytes - 99pct: 65368.000 bytes - 99.9pct: 65465.000 bytes - max: 65465.000 bytes --- Ack received rate: 69.33 ack/s --- Failed messages: 0 --- Pending messages: 15 [persistent://public/default/-partition-0] [-175-224582] Pending messages: 88 --- Publish throughput: 103.66 msg/s --- 0.76 Mbit/s --- Latency: med: 2735.000 ms - 95pct: 5846.000 ms - 99pct: 5998.000 ms - 99.9pct: 6753.000 ms - max: 6753.000 ms --- BatchSize: med: 3.000 - 95pct: 59.000 - 99pct: 69.000 - 99.9pct: 69.000 - max: 69.000 --- MsgSize: med: 3066.000 bytes - 95pct: 57746.000 bytes - 99pct: 65327.000 bytes - 99.9pct: 65510.000 bytes - max: 65510.000 bytes --- Ack received rate: 110.80 ack/s --- Failed messages: 0 --- Pending messages: 88 2025-07-04 124036.833 [pulsar-timer-6-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://public/default/-partition-2] [-175-224582] Pending messages: 0 --- Publish throughput: 105.25 msg/s --- 0.78 Mbit/s --- Latency: med: 4.000 ms - 95pct: 252.000 ms - 99pct: 261.000 ms - 99.9pct: 430.000 ms - max: 961.000 ms --- BatchSize: med: 3.000 - 95pct: 63.000 - 99pct: 69.000 - 99.9pct: 69.000 - max: 69.000 --- MsgSize: med: 2878.000 bytes - 95pct: 59276.000 bytes - 99pct: 65241.000 bytes - 99.9pct: 65469.000 bytes - max: 65528.000 bytes --- Ack received rate: 105.25 ack/s --- Failed messages: 0 --- Pending messages: 0 have 4 broker running but still the topic is not distributed .partition1 and partition 0 are running on same broker . topic unload nor namespace unload is not redistributing topics across brokers .am having threshold shredder and running on 2.10.x pulsar .

Hey, curious if anybody who is familiar with Entry Filters has ideas on why I’m observing higher bookie CPU while the filter is active, despite the fact that the filter runs on the broker: https://github.com/apache/pulsar/discussions/24493

We want external partners be able to consume topics. These partners are outside our corporate networking boundaries and our policies currently prevent using the Pulsar or Kafka protocol over the public internet. I found Pulsar Beam [1] which seem to do the job (but looks abandoned). Who is using this? Or has a different approach? [1] https://github.com/kafkaesque-io/pulsar-beam Thanks, Jeroen

Hello everyone. I create a jdbc sink for mysql with this configuration

tenant: public
namespace: default
name: jdbc_sink_pulsar_to_mysql_temp
archive: connectors/pulsar-io-jdbc-sqlite-4.0.0.nar
inputs:
  - <persistent://public/default/temp_schema>
configs:
  jdbcUrl: "jdbc:<mysql://mysql:3306/mqtt_db>"
  userName: "user1"
  password: "1234567890"
  tableName: "pulsar_to_db_temp"
  insertMode: INSERT
  key: "message_id"  
  nonKey: "temperature,timestamp,pulsar_timestamp"

and i mount the connector under the connectors folder like so

- ./pulsar-mysql/pulsar-io-jdbc-sqlite-4.0.0.nar:/pulsar/connectors/pulsar-io-jdbc-sqlite-4.0.0.nar

but i get this error

ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Sink open produced uncaught exception:
java.sql.SQLException: No suitable driver found for jdbc:<mysql://mysql:3306/mqtt_db>
        at java.sql.DriverManager.getConnection(Unknown Source) ~[java.sql:?]
        at java.sql.DriverManager.getConnection(Unknown Source) ~[java.sql:?]
        at org.apache.pulsar.io.jdbc.JdbcAbstractSink.open(JdbcAbstractSink.java:97) ~[pulsar-io-jdbc-core-4.0.0.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupOutput(JavaInstanceRunnable.java:1080) ~[?:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setup(JavaInstanceRunnable.java:263) ~[?:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:313) ~[?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

Am I using the wrong connector? Or am I missing a configuration

Good afternoon! I have 4.0.4 of Pulsar deployed and am experiencing an issue where Bookkeeper is leaving some orphaned ledgers around. At first I had this happening while offloading to S3 - my general configuration was to offload every 15 minutes, with a deletionLag of 0. (I've tried tweaking the offload settings many times, but my bookie PVCs kept filling due to these orphaned ledgers). Thinking it might be an issue with offloading, I switched to disabling offloading, increasing ledger volumes, and just setting a shorter retention time. When I look in the

/pulsar/data/bookkeeper/ledger0/current

I see files .log files days older than my retention period. When researching this issue, I came across [PCK](https://docs.streamnative.io/private-cloud/v1/tools/pck/pck-overview) which seems to confirm this issue, as it aims to mitigate the problem. This does not look to be publicly available though - are there any alternative solutions? Thanks in advance!

context • ordering required • have partitioning for throughput • using pulsar client (non-reactive) • listener threads are getting shared across internal queue in a blocking fashion • mostly io ques: in order to have higher throughput i can increase listener threads so that internal queues processing do not block each others. partitions and topics will increase and therefore there’s not end to it. How can pulsar reactive client help with this? i want high throughput with less number of thread. how does it overcome native pulsar client listeners blocking behaviour? pipelining i am aware of but it seems conflicting with partitions

Hello everyone I have a Pulsar cluster with 5 brokers (each broker is on the same server as a bookie) and 3 ZooKeeper servers. Every day during the backup process, the bookie servers shut themselves down. What can I do to prevent this? Also, is it necessary to back up the bookies? Would backing up only the ZooKeepers be sufficient?

How reliable is the dispatch data (values set, not the dispatch feature itself) when the brokers restart? • On the application side, the consumers check for specific values and stop if the are set. I'm currently using the dispatch rate on the topic with pre-defined value to "pause" the consumer (stop them from reading any new messages) • The dispatch rate is set (done with pulsar-admin CLI) at

1

for the

msg-dispatch-rate

and

315600000

for the

dispatch-rate-period

, which essentially lower as much as possible the message throughput (I'm aware we can't actually pause message throughput with the dispatch rate). But I noticed that the dispatch rate values returned by the admin client, on the application side, when the brokers restart, are empty as if they were not set. The consequence is that it unpauses the consumers, only to re-pause then after the brokers are done restarting. I just wanted to know if this is an expected behavior? I was assuming that this type of data, like topic properties was some sort of distributed data (from zookeeper) and was not impacted by brokers restart

I am currently having issue with pulsar configuration on my our kubernetes (exoscales sks). After the installation and I check the pods only the zookeeper is running and the toolset Others where stuck at init 0/2 podinitializing. I also set all replica to one and disabled monitoring. the environment is set to use less that 3 nodes too. All storage class where assigned as well , all bounded successful.

pulsarlogo 📣 [ANNOUNCE] Apache Pulsar 3.0.13, 3.3.8 and 4.0.6 released 📣 pulsarlogo For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: 3.0.13: https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.13/ 3.3.8: https://pulsar.apache.org/release-notes/versioned/pulsar-3.3.8/ 4.0.6: https://pulsar.apache.org/release-notes/versioned/pulsar-4.0.6/ (Current LTS release) Although 3.3.8 was released, please notice that support for 3.3.x has already ended and no new releases are planned.

Hi everyone, I’ve been working with Apache Pulsar recently and truly admire how far it has come. I’m exploring ways to build tools and services around Pulsar in 2025, focused on growing adoption in new regions and smaller teams. From your experience, how is the demand for Pulsar evolving in 2025 compared to previous years? Are you seeing more interest from companies moving away from Kafka or adopting Pulsar from scratch? I’d love to understand where the ecosystem is heading. Any trends, use cases, or growth signals would be super helpful to know as someone building on top of Pulsar. Thanks in advance

Hi does anyone know how to extract the data from bookkeeper? I need to query the message, when i use version 3 there is trino for me to use SQL But version 4, trino is removed