Hello! I am trying to understand what are the necessary conditions to achieve effectively once semantics when getting data to Pulsar topic from "external world". I captured my understanding in this github discussion: https://github.com/apache/pulsar/discussions/24605 Trying to get some feedback if my understanding is correct. What's the best way to get some feedback on this? Greets

Good morning! I have a question regarding geo-replication: I have a two cluster geo-replication set up deployed via kubernetes. when running a producer on cluster A for the first time (using auto topic creation - did not manually create it) my producer successfully runs and sends the message on the cluster A service url. before starting the consumer on cluster B on that same topic, I wanted to check the topic stats on cluster B, but it says the topic doesn't exist. Once I start the consumer, the consumer on the cluster B service url, it receives the messages successfully. But I thought topics were supposed to be replicated across clusters? Is this an error in my geo-replication set-up, or is this expected behavior? Would enabling replicated subscriptions help this issue?

Hi Guys. I recently created one issue. Would appreciate any help on the same: https://github.com/apache/pulsar/issues/24755 Thank you 🙏

Hello, I'd like to know if there is a plan to upgrade the current flink pulsar connector to be supported by new flink versions, the official connector is compatible onlly with flink 1.18. The problem is that I can see that the oficial github repo is abandoned, no one is checkin' the PRs, a fix was marged many months ago and there is no new build image about it. Why?

Has Debezium been moved to the latest version in Pulsar? We are using Oracle, and the current version of Debezium always fails in case of DDL changes, making it rarely work because of the parsing error. Anybody has a solution to this?

Hi. Why does Pulsar Sink crash if it is created with

EFFECTIVELY_ONCE

guarantee and the sink calls the

record.fail()

?

👋 happy friday! I filed an issue yesterday, seems like there might be a memory leak with pulsar key-based batching in the pulsar go client producer. Posting in #C5Z4T36F7 to raise awareness, thank you! 🙇 https://github.com/apache/pulsar-client-go/issues/1423 🤔

Hi, I am now setting up a pulsar cluster which utilized Oxia as the metadata store, and now I am facing the following issue in my Oxia cluser

{
  "level": "warn",
  "time": "2025-09-20T07:01:17.961738475Z",
  "component": "public-rpc-server",
  "error": {
    "error": "oxia: failed to append to wal: 535046 can not immediately follow 459362: oxia: invalid next offset in wal",
    "kind": "*errors.withStack",
    "stack": null
  },
  "namespace": "bookkeeper",
  "peer": "10.194.131.14:52392",
  "shard": 6,
  "message": "Write stream has been closed by error"
}

I did search the related topic in the Oxia github page, but found no issues/discussions. Does anyone have had facing the similar issues before, or have any clue about how to resolve it?

Having a bookkeeper replicaCount at 4, what would be a most natural Ensemble-Qw-Qa setting, if I want to tolerate one of the replicas going down even for a considerable amount of time (node outage, etc.). I believe E should be less than replicaCount (to tolerate one replica going down), while best to have E=Qw (avoid issues with lack of consistency). So I guess 3-3-3 or 3-3-2 would be best. What configurations would be recommended?

I am seeing pulsar perf producer is showing 50K msg/sec rate where as grafana dashboard is showing 12K. I have 4 paritions, by any channce it is showing single parition message rate?

📣 [ANNOUNCE] Apache Pulsar 3.0.14, 3.3.9, 4.0.7 and 4.1.1 released 📣 For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at: • 3.0.14: https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.14/ (previous LTS release, support until May 2026) • 3.3.9: https://pulsar.apache.org/release-notes/versioned/pulsar-3.3.9/ (support has already ended in December 2024) • 4.0.7: https://pulsar.apache.org/release-notes/versioned/pulsar-4.0.7/ (Current LTS release) • 4.1.1: https://pulsar.apache.org/release-notes/versioned/pulsar-4.1.1/ (Latest release) Please check the release notes for more details.

Hi, Seems to me an answer to my question could be quite useful to a lot of people new to Pulsar that try to design around it with constraint of effectively one delivery semantics. Is there anything I can do to get some attention to it from Pulsar community?

Hi, I'm running a Pulsar 4.0.2 Cluster (3 broker, 3 bookies, 3 zookeeper) with partitioned topics and I have two problems: 1. The filesystem usage on the bookies doesn't seem to go down (bk1: 100GB, bk2: 400GB, bk3: 100G). I already set a retention policy on my main namespace (2weeks, 10GB) and the metriks in Grafana report the correct topic sizes (100GB storage size and ~30GB backlog size). I get the usage on bk1 and bk3, but not the 400GB on bk2... 2. The bookie service on bk2 stops frequently with an error "io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 2147483648, max: 2147483648)". I can't find the setting where I can control the limit for this memory setting. I already increased the jvm allocations in the pulsar_env.sh those don't seem to correlate.. Thanks!

We've just released Apache Pulsar Helm Chart 4.3.0 🎉 The official source release, as well as the binary Helm Chart release, are available at https://www.apache.org/dyn/closer.lua/pulsar/helm-chart/4.3.0/?action=download The helm chart index at https://pulsar.apache.org/charts/ has been updated and the release is also available directly via helm. Release Notes: https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-4.3.0 Docs: https://github.com/apache/pulsar-helm-chart#readme and https://pulsar.apache.org/docs/helm-overview ArtifactHub: https://artifacthub.io/packages/helm/apache/pulsar/4.3.0 Thanks to all the contributors who made this possible.

Hello, I want to use pulsar with .Net but the librsry dotpulsar is too basic without many features, so I saw this client matrix https://pulsar.apache.org/client-feature-matrix/#consumer Where there is a column called .Net (C#,F#,VB), is this other library?

Hi Team, We are currently operating a clustered environment consisting of two instances, running Apache Pulsar version 4.1.1 . During load testing, with "brokerDeduplicationEnabled= true", we have encountered issues related to message loss. For your reference and to support further analysis, I have attached the broker configuration file. And the MessageId corresponding to an unpublished message is. { "partition": -1, "ledgerId": -9223372036854800000.0, "batchIndex": -1, "entryId": -9223372036854800000.0 } DotPulsar version 4.3.2 is being used as the client library. We would appreciate your assistance in investigating this issue and welcome any insights or recommendations you may have.

Subject: Bookie crashes with -25 error when enabling StreamStorageLifecycleComponent (Pulsar 4.1.1) Hi everyone 👋 — I’m setting up a small Pulsar 4.1.1 cluster and running into an issue when enabling stream storage. Cluster Setup I have 3 VMs: Host CPUs RAM Storage Roles 10.0.1.74 8 16GB 500GB SSD ZooKeeper + Bookie + Broker 10.0.1.75 8 16GB 400GB SSD ZooKeeper + Bookie + Broker 10.0.1.91 12 64GB 100GB SSD ZooKeeper Cluster Init Command bash bin/pulsar initialize-cluster-metadata \ --cluster dev-cluster-1 \ --metadata-store zk10.0.1.742181,10.0.1.752181,10.0.1.912181 \ --configuration-metadata-store zk10.0.1.742181,10.0.1.752181,10.0.1.912181 \ --web-service-url http://10.0.1.74:8080,10.0.1.75:8080 \ --web-service-url-tls https://10.0.1.74:8443,10.0.1.75:8443 \ --broker-service-url pulsar://10.0.1.74:6650,10.0.1.75:6650 \ --broker-service-url-tls pulsar+ssl://10.0.1.74:6651,10.0.1.75:6651 Bookie Config (relevant parts) properties storageserver.grpc.port=4181 dlog.bkcEnsembleSize=2 dlog.bkcWriteQuorumSize=2 dlog.bkcAckQuorumSize=1 storage.range.store.dirs=data/bookkeeper/ranges storage.serve.readonly.tables=false storage.cluster.controller.schedule.interval.ms=30000 Issue When I run Pulsar in stateless mode, ZooKeeper, BookKeeper, and Brokers all start fine. But when I enable: properties extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent both Bookies crash shortly after startup with a BKTransmitException and error -25. Printed using ChatGPT to PDF, powered by PDFCrowd HTML to PDF API. 8/9 Excerpt from logs kotlin Caused by: org.apache.distributedlog.exceptions.BKTransmitException: Failed to open ledger handle for log segment ... : -25 What I’ve tried Verified ZooKeeper quorum and BookKeeper ledger directories Cleaned /data/bookkeeper and restarted the cluster Ensured the ensemble/write/ack quorum configs match cluster size Question Has anyone successfully enabled the stream storage component (StreamStorageLifecycleComponent) on Pulsar 4.1.1? What does the -25 (BKTransmitException) usually indicate in this context — ZK metadata corruption, missing ledger, or a config mismatch? Any guidance or example configurations for Pulsar 4.x stream storage clusters would be greatly appreciated

Hi Team 👋 I'm investigating an issue related to the Pulsar broker configuration parameter

delayedDeliveryTrackerFactoryClassName

. We initially used

InMemoryDelayedDeliveryTracker

, (because we where using version 2.7 of pulsar) which caused acknowledged delayed messages to be reprocessed after a broker restart likely due to its state stored only in memory. Given our high message volume (millions), this behavior is problematic. A screenshot is available showing the lag escalation following a broker restart. We're generating delayed messages out of sequence, resulting in gaps within the acknowledged message stream. This causes non-contiguous ranges of messages to be marked as deleted or eligible for deletion. In our screenshot, the value of

nonContiguousDeletedMessagesRanges

is 16833. To mitigate this following the upgrade of pulsar to the version 4.0.4, we updated the broker config to use

org.apache.pulsar.broker.delayed.BucketDelayedDeliveryTrackerFactory

, which should persist delayed delivery metadata to disk via BookKeeper ledger buckets. However, after switching to the bucket-based tracker, we're still seeing the same behavior post-restart. A few observations and questions: • I checked the

pulsar_delayed_message_index_loaded

metric and noticed that messages are still being loaded into memory, while

pulsar_delayed_message_index_bucket_total

remains at zero. Is this expected? Shouldn’t the bucket tracker be persisting and loading from disk? • Are there additional broker settings required to fully enable bucket-based delayed delivery tracking? For example: ◦ Do we need to explicitly configure

delayedDeliveryTrackerBucketSize

or

delayedDeliveryMaxNumBuckets

? ◦ Is there any dependency on topic-level settings or namespace policies that could override the broker-level tracker configuration? ◦ Could other settings interfere with delayed message persistence? Any insights or guidance would be greatly appreciated. Thanks for your help!

Hi team: I DO REALLY NEED THE HELP RIGHT NOW. One of my bookie node in the production cluster are stuck in the CrashLoopBackOff status; after checking the logs, I find out that it is because the node cannot successfully access the lock from the metadata server (which we use Oxia in our cluster). I have tried to restart the Oxia cluster, since I saw some error messages saying that the oxia cannot append the WAL to the stream (I were facing the similar situation last month, and the restarting Oxia trick did work for me). Unfortunately, the bookie node still cannot get it's own lock after restarting the Oxia. I REALLY NEED SOMEONE THAT CAN HELP ME OUT WITH THE ISSUE 😢

we seeing Oxia as a metadata storage system crashing repeatedly for us after 20 minutes of use with WAL corruption as described here https://github.com/oxia-db/oxia/issues/772 did anyone experienced something similar and can share workarounds?

Hi all... I'm debugging an issue where we're using Pulsar with non-persistent topics, and in this case, we have only 1 consumer. The data rate is low -- ~5000 msgs/sec, 1kb each -- but we see some small amount of drops. Normally, we have much larger scale and we can successfully run with 50 consumers at 3 million msgs/sec with 0 drops. So it's odd we have drops here where CPU usage on every component is quite low. The Pulsar consumer receive queue is set to 50k and never gets low. I've also taken our software out of it and replaced our consumers with the sample Java consumer code from the docs. There, we just throw away the message (no

println()

or other work per message). Again, CPU usage is under 10% for all components, but I see the same small drops. I started debugging and found Pulsar is dropping because the Netty connection's

.isWritable()

returns false and this causes Pulsar to immediately drop. This "Returns

true

if and only if the I/O thread will perform the requested write operation immediately", meaning there is room available in Netty's ChannelOutboundBuffer. I found that if I increase the Netty low/highWaterMarks, the drops go away, but it's not possible without a code change to Pulsar broker. I'm looking for any suggestions on different configurations I can try. Thanks!!

We are using Apache Spark for batch processing and are now planning to extend the same for Stream Processing. I was checking the stream-native pulsar connector and was wondering does it supports Continuous Trigger or RealTime trigger like Databricks? The idea is to have sub-milliseconds end-to-end latency. Is this at all possible with Pulsar <> Spark?

📣 Dekaf UI is now open-source software licensed under Apache 2.0: https://github.com/visortelle/dekaf Please try, rise bugs, feature requests, and ask questions on GitHub. Thank you!

Hi all, is there a good guide for migrating from zookeeper to oxia? My searches are coming up empty. Thanks 😄

Unfortunately, there isn’t really an easy way to do that. The main issue is that the data is stored in very different formats, so it can’t just be copied.

Hi all, I have a question about this issue: https://github.com/apache/pulsar-client-go/issues/1297. We don’t specify consumers' name, and I understand that Pulsar will create a unique name in that case. However, if the consumer name is unique, shouldn’t the producer name (

..<subscription>-<consumerName>-DLQ

) be as well, since it uses the consumer name in the producer’s name? Will consumer stop consuming message if this happen? We are using Pulsar client 4.0.0 where producer name constructed as:

.producerName(String.format("%s-%s-%s-DLQ", this.topicName, this.subscription, this.consumerName))

Hi everyone We’re using Pulsar with strict schema governance - each namespace has

schemaValidationEnforced=true

and

isAllowAutoUpdateSchema=false

(only under an approved process), so only admins can push schemas. Here’s the issue: when a consumer is configured with a

DeadLetterPolicy

and a message fails too many times (or is negatively acknowledged repeatedly), the client will publish the message to a dead-letter topic (default name

<topic>-<subscription>-DLQ

) after the redelivery threshold. That topic doesn’t necessarily exist ahead of time (unless created before), so when it’s first used it may trigger topic creation and/or schema registration. Because our namespace forbids auto schema updates and enforces schemas, this can fail - the consumer isn’t authorized to register the schema for the DLQ topic. To work around this, we’re creating a separate namespace (e.g.,

<namespace>-dlq

) where: •

isAllowAutoUpdateSchema=true

•

schemaValidationEnforced=false

• so consumers can safely publish DLQ messages without schema conflicts. Is this the recommended approach? Is there a cleaner way to allow DLQ schema creation while keeping production namespaces locked down? Any official guidance or community best practices would be really appreciated 🙏 Thanks!

hello everybody, I have a reproducible test case that demonstrate that pulsar 2-way geo replication doesn't support exclusive access producers. If this is a limitation made by design, I am not sure what benefit that could bring. Otherwise, if it's not a limitation made by design, I believe that it should be resolved to strengthen and improve the pulsar geo replication feature. I have opened this issue 24914. Will appreciate if someone can take a look and suggest whether it makes sense or not.

Hi Folks, I was exploring Apache Pulsar functions. From the documentation, we seem to use Apache Bookkeeper as the state storage interface. Couple of questions on the state storage for pulsar functions: 1\ Do we have any upper limit on the storage allocated for the pulsar function state storage? 2\ Do we have a tiered storage paradigm even for the pulsar function state storage similar to the event stream storage? TIA.

I’m facing a problem where the consumer internal DLQ producer can’t connect due to a conflict with another producer with the same name. This issue was fixed in version 4.1.0, but it requires Java 17, which is not feasible for us at the moment (still with java8 ) I want to implement a short-term workaround to detect this issue and recreate the consumer with a different name, which, in theory, should resolve the conflict. The pulsar client implementation, however, seems to hide the entire DLQ handling, with exceptions not thrown or propagated back to external code. Is there a way to archive what I need here?