Hi all... I'm debugging an issue where we're using Pulsar with non-persistent topics, and in this case, we have only 1 consumer. The data rate is low -- ~5000 msgs/sec, 1kb each -- but we see some small amount of drops. Normally, we have much larger scale and we can successfully run with 50 consumers at 3 million msgs/sec with 0 drops. So it's odd we have drops here where CPU usage on every component is quite low. The Pulsar consumer receive queue is set to 50k and never gets low. I've also taken our software out of it and replaced our consumers with the sample Java consumer code from the docs. There, we just throw away the message (no

println()

or other work per message). Again, CPU usage is under 10% for all components, but I see the same small drops. I started debugging and found Pulsar is dropping because the Netty connection's

.isWritable()

returns false and this causes Pulsar to immediately drop. This "Returns

true

if and only if the I/O thread will perform the requested write operation immediately", meaning there is room available in Netty's ChannelOutboundBuffer. I found that if I increase the Netty low/highWaterMarks, the drops go away, but it's not possible without a code change to Pulsar broker. I'm looking for any suggestions on different configurations I can try. Thanks!!

We are using Apache Spark for batch processing and are now planning to extend the same for Stream Processing. I was checking the stream-native pulsar connector and was wondering does it supports Continuous Trigger or RealTime trigger like Databricks? The idea is to have sub-milliseconds end-to-end latency. Is this at all possible with Pulsar <> Spark?

📣 Dekaf UI is now open-source software licensed under Apache 2.0: https://github.com/visortelle/dekaf Please try, rise bugs, feature requests, and ask questions on GitHub. Thank you!

Hi all, is there a good guide for migrating from zookeeper to oxia? My searches are coming up empty. Thanks 😄

Unfortunately, there isn’t really an easy way to do that. The main issue is that the data is stored in very different formats, so it can’t just be copied.

Hi all, I have a question about this issue: https://github.com/apache/pulsar-client-go/issues/1297. We don’t specify consumers' name, and I understand that Pulsar will create a unique name in that case. However, if the consumer name is unique, shouldn’t the producer name (

..<subscription>-<consumerName>-DLQ

) be as well, since it uses the consumer name in the producer’s name? Will consumer stop consuming message if this happen? We are using Pulsar client 4.0.0 where producer name constructed as:

.producerName(String.format("%s-%s-%s-DLQ", this.topicName, this.subscription, this.consumerName))

Hi everyone We’re using Pulsar with strict schema governance - each namespace has

schemaValidationEnforced=true

and

isAllowAutoUpdateSchema=false

(only under an approved process), so only admins can push schemas. Here’s the issue: when a consumer is configured with a

DeadLetterPolicy

and a message fails too many times (or is negatively acknowledged repeatedly), the client will publish the message to a dead-letter topic (default name

<topic>-<subscription>-DLQ

) after the redelivery threshold. That topic doesn’t necessarily exist ahead of time (unless created before), so when it’s first used it may trigger topic creation and/or schema registration. Because our namespace forbids auto schema updates and enforces schemas, this can fail - the consumer isn’t authorized to register the schema for the DLQ topic. To work around this, we’re creating a separate namespace (e.g.,

<namespace>-dlq

) where: •

isAllowAutoUpdateSchema=true

•

schemaValidationEnforced=false

• so consumers can safely publish DLQ messages without schema conflicts. Is this the recommended approach? Is there a cleaner way to allow DLQ schema creation while keeping production namespaces locked down? Any official guidance or community best practices would be really appreciated 🙏 Thanks!

hello everybody, I have a reproducible test case that demonstrate that pulsar 2-way geo replication doesn't support exclusive access producers. If this is a limitation made by design, I am not sure what benefit that could bring. Otherwise, if it's not a limitation made by design, I believe that it should be resolved to strengthen and improve the pulsar geo replication feature. I have opened this issue 24914. Will appreciate if someone can take a look and suggest whether it makes sense or not.

Hi Folks, I was exploring Apache Pulsar functions. From the documentation, we seem to use Apache Bookkeeper as the state storage interface. Couple of questions on the state storage for pulsar functions: 1\ Do we have any upper limit on the storage allocated for the pulsar function state storage? 2\ Do we have a tiered storage paradigm even for the pulsar function state storage similar to the event stream storage? TIA.

I’m facing a problem where the consumer internal DLQ producer can’t connect due to a conflict with another producer with the same name. This issue was fixed in version 4.1.0, but it requires Java 17, which is not feasible for us at the moment (still with java8 ) I want to implement a short-term workaround to detect this issue and recreate the consumer with a different name, which, in theory, should resolve the conflict. The pulsar client implementation, however, seems to hide the entire DLQ handling, with exceptions not thrown or propagated back to external code. Is there a way to archive what I need here?

Hey! I'm evaluating migration options for moving our Pulsar stacks from EC2 to Kubernetes environments, where the requirement is to minimize the service downtime. So far so good I learned a lot about the geo-replication, which could be used by me, but I'm wondering what the story is behind the PIP-188 https://github.com/apache/pulsar/issues/16551 . AFAIk the work has been delivered, but I can't find instructions on how to leverage and try this solution. The list of features is impressive

Publish ordering guarantee
Consumer ordering guarantee
Incoming replicator ordering guarantee
Outgoing replicator ordering guarantee with the topic unavailability tradeoff
Auto resource creation (tenant, namespace, partitioned-topic, subscriptions) in a green cluster
Auto topic deletion after migration successfully completed for a topic
Enable migration at cluster level or per namespace level
Stats to show topic's migration state

But the thing is that due to missing configuration steps, it's hard to test this feature. Can someone explain how to start with that?

<URGENT> Hi, there are many .log files under the /data/bookie1/ledgers/current directory, which occupy a large amount of space. How can I clean them up? For temporary fix can i manually delete these log files ?

Pulsar Flink Connector side we don't have any support from Pulsar side, currently?

Hi Team ,On pulsar bookkeeper server under /data/bookkeeper/ledgers/current there are old .log files which are older like 2 years more , we set retention in broker for acknowlege messages to 60 minutes . But still the old log files are not getting delete , ledger is the most consuming one which is filling disk space Checked https://apache-pulsar.slack.com/archives/C5Z4T36F7/p176278302931373

📣 [ANNOUNCE] Apache Pulsar 3.0.15, 4.0.8 and 4.1.2 released 📣 For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: • 3.0.15: https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.15/ (previous LTS release, support until May 2026) • 4.0.8: https://pulsar.apache.org/release-notes/versioned/pulsar-4.0.8/ (Current LTS release) • 4.1.2: https://pulsar.apache.org/release-notes/versioned/pulsar-4.1.2/ (Latest release) Please check the release notes for more details.

Hi everyone, has anyone experience

504 Gateway Timeout

from pulsar clients in a production cluster ? We are currently experiencing timeout of proxies from time to time on multiple clusters with a HTTP

504

, exception message is

SSL BAD PACKET LENGTH

. It looks like an issue between proxy - broker connection pool, but cannot yet prove it. We're running

4.1.0

We have to reboot proxies to fix the issue for now

What's the technical reason between having journal/ledger on same nvme versus having two separate drives, one for journal and one for ledgers?

There are several reasons why you should have the journal and ledger disks on separate physical volumes, including performance. But the primary reason is that they serve two different purposes. The journal disk is used for short-term storage of the messages before they are indexed and written to the ledger disk. The journal disk provides data durability guarantees in the event of a failure, the bookie can recover and load the messages from the journal disk. However, if the journal disk fails, Pulsar will continue to operate. So separating them eliminates a single point of failure from the storage layer as well.

Hi all 👋 I have a question about per-key scheduling behavior in Pulsar. I need strict ordering and exclusivity per

sensor_id

, but I don’t want long-lived key to consumer stickiness. Instead, I’m trying to achieve this logic:

If no consumer is currently processing

sensor_id = X

, then the next message for that sensor should be assigned to the next available consumer (round-robin or least-loaded).

All while preserving ordering and ensuring no two consumers ever process the same key concurrently.

KeyShared

ensures ordering and exclusivity, but it uses stable key-range hashing, so a key stays with one consumer until that consumer dies. Is there any Pulsar pattern, config, or upcoming feature that supports dynamic per-message key assignment instead of sticky key-range ownership? Or is this fundamentally outside Pulsar’s delivery semantics? Thanks! 🙏

We already use a shared subscription with

KeyShare

option, since we do want messages to be processed in order per Key (this is something our design requires)

Hi guys, how's everyone doing?

I had a scenario with me, in which I have microservices A and B (both are multi pod deployments), A does some computation and updates its database, then it needs to let all the pods of B know about the updates, which are some basic configs. how can this be done so the system is scalable? and I don't want service B to use much resources coz it is already resource constrained... I have checked the following approaches which are allowed in my company: 1. through kafka: all pods of service B will have a different consumer group, and they all subscribe to the same topic into which A would push its updates 2. through pulsar: similar to kafka, and I also got to know that pulsar is more suitable for these kinds of fan out scenarios. how exactly I so have some idea, but don't know this exactly 3. through zookeeper: this is more preferred by my seniors who have some experience using it. They have claimed that the approach using zk would use the least resources, and would suit for usage in service B. which approach should I use given the constraints? and if pulsar, then how exactly is its subscription model better than kafka?

hey pulsarers! is there any chance that this issue gets merged? https://github.com/apache/pulsar/issues/24914

We've just released Apache Pulsar Helm Chart 4.4.0 🎉 The official source release, as well as the binary Helm Chart release, are available at https://www.apache.org/dyn/closer.lua/pulsar/helm-chart/4.4.0/?action=download The helm chart index at https://pulsar.apache.org/charts/ has been updated and the release is also available directly via helm. The main highlights of this release are the upgrade of the default Pulsar version to 4.0.8 and the Helm chart's integration with Dekaf UI. Dekaf is a web-based UI for Apache Pulsar, licensed under Apache 2.0 (GitHub: https://github.com/visortelle/dekaf). Thanks to @Kiryl Valkovich for this great contribution to the Apache Pulsar community! Release Notes: https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-4.4.0 Docs: https://github.com/apache/pulsar-helm-chart#readme and https://pulsar.apache.org/docs/helm-overview ArtifactHub: https://artifacthub.io/packages/helm/apache/pulsar/4.4.0 Thanks to all the contributors who made this possible.

Hello, I am trying to update the bearer token that a function is using in the Kubernetes runtime with a REST API. I am able to use the CLI

bin/pulsar-admin functions update --tenant <TENANT> --namespace <NS> --name example-test-function --update-auth-data

to perform this functionality. Is there a way to do this with the functions REST API?

After update client from 4.0.0 to 4.0.7 (to include change to resolve issue where DLQ producer name conflict) we got exception:

org.apache.pulsar.client.api.PulsarClientException$FeatureNotSupportedException: The feature of getting partitions without auto-creation is not supported by the broker. Please upgrade the broker to version that supports PIP-344 to resolve this issue.

Looking at the code, i see something like

useFallbackForNonPIP344Brokers

it seems like from 4.0.7 no longer support falling back? what version from 4.0.0 to 4.07 that have the DLQ producer name conflict fix but still supporting fallback for non PIP-344 broker?

Hello, We are experiencing an issue on some topics (persistent) in our cluster in production where some messages are consumed with a big delay after they are produced (45h or more). It seems that when the issue happens, it's located on specific partition(s) for a given topic. I think (maybe) that this issue occurs on topics that don't have much load, we're talking about a few messages a day because the heavy used topic don't seem to show that behavior, but can't be too sure as of right now. We also have more consumers (Pod/instance) for the "big" topics and they are all using a

shared

subscription with

16

partitions. We are using Pulsar

4.0.6

(via helm chart), our applications are using the Go client (commit

ed7d4980034871e1db28770576151c4c05c7d0ea

) I noticed this behavior a few times already. I know that it happened once when the brokers were restarted (rollout used to move workload to other k8s nodes, no config change there), I'm not sure if that's the case when applications (Go apps) restart and I'm still trying to see if this is a server or client problem. Is there anything that could potentially trigger that behavior? I'm thinking maybe partition discovery in the Go client, or an issue related to the partition consumer as I this there is an issue on Git: https://github.com/apache/pulsar-client-go/issues/1426 I'm not seeing anything obvious that could fix the issue in the release logs of

4.0.7

or

4.0.8

, or similar issues open on Git. Regardless, any feedback is welcome. Thanks for the help

Hello I am unable to deploy a Debezium source Connector. I was able to do so without any issues on my staging cluster, but not on the production cluster. Everything appears to be identical. Here is the error I am encountering:

2025-11-25T11:18:25,109+0000 [pulsar-web-41-5] ERROR org.apache.pulsar.functions.worker.rest.api.SourcesImpl - Failed process Source debezium/XXXXXX package: 
org.apache.distributedlog.exceptions.BKTransmitException: Failed to transmit entry : -6
------
Caused by: org.apache.distributedlog.exceptions.BKTransmitException: Failed to write to bookkeeper; Error is (-6) Not enough non-faulty bookies available: -6

All my bookies are healthy Do you have any ideas ?

Hello! I’m new to Pulsar and I’d like to build a Map-Reduce like pipeline using Pulsar Functions in Python: producer -> (submit raw messages without key) -> Topic A -> (Shared subscription) -> “Key Function” to assign key for each record -> (submit messages with keys) -> Topic B -> (key_shared subscription) -> “Compute Function” to process records The key requirements are: 1. The records of the same key should be processed in sequence and in micro batch 2. Different keys should be processed in parallel 3. Low end2end latency: around 100ms 4. High throughput: 1 million messages / second Questions: 1. Can “Compute Function” process a (micro) batch of records at a time. It seems like Pulsar Function doesn’t support batch input? 2. If batch input is not supported, is it a common/good practice to buffer the inputs inside function instance? What's the latency like to persist every record (1KB) to the BookKeeper? 3. Is it possible to create multiple threads within a function instance to handle different keys? Is it anti-pattern? Thanks!

Hello. We are experiencing an issue with subscription replication. Pulsar version 4.0.7. We have two clusters: a primary production cluster and a secondary backup stand-by cluster. Subscription replication is enabled. At some point, a subscription stops replicating, and backlog begins to accumulate on the remote cluster (topic replication itself continues normally). Around a hundred topics are being replicated simultaneously, but the issue occurs only on one or a few of them. In some cases, the only workaround is to disable replication for the affected topic, clear the backlog on the remote cluster, and then re-enable replication. Sometimes a portion of updates still manages to replicate. There are no errors or warnings in the logs. What could be causing this issue? What additional information would be useful? In the topic stats for the subscription, the status shows replicated: true. Thanks!