So now you are getting more fsyncs, which will slow things down

Yeah we need consistency

We are ok with the higher latency

Agreed. this was a necessary bug fix.

We just wanted to understand why suddenly out of the blue we got crazy performance boost in 3.0.0 😄

Mystery solved

Thanks a lot @David K for helping with this. 🙏

I am running some unit tests after starting a Docker container via https://github.com/ory/dockertest. However, the Apache Pulsar standalone node is getting killed with exit code 137. Can someone please help me with the environment variables/command line flags needed to constrain memory used by Pulsar? I tried asking @/RunLLM here but it didn't work. The run options I was running with are as follows:

dockertest.RunOptions{
		Repository:   "apachepulsar/pulsar",
		Tag:          "4.0.4",
		Cmd:          []string{"bin/pulsar", "standalone"},
		ExposedPorts: []string{o.BinaryPort, o.RESTPort},
		PortBindings: map[docker.Port][]docker.PortBinding{
			"6650": {{HostIP: "0.0.0.0", HostPort: o.BinaryPort}},
			"8080": {{HostIP: "0.0.0.0", HostPort: o.RESTPort}},
		},
	}

Hey everyone! I'm currently looking for a solid Ruby on Rails client library for Apache Pulsar. From what I’ve found so far, it seems like there aren’t many options available. Has anyone here worked with Pulsar in a Rails environment or have any recommendations or experiences to share?

Is the command

pulsar-admin broker-stats load-report

compatible with the extensible load balancer? When running the command from a broker, I get an exception We are on pulsar

4.0.4

(we just migrated from

4.0.3

and the modular load balancer where I was using the command successfully) and this is the config we set ourselves inside the helm chart

loadManagerClassName: "org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl"
    loadBalancerLoadSheddingStrategy: "org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder"

Command return is:

pulsar-broker-2:/pulsar$ pulsar-admin broker-stats load-report

 --- An unexpected error occurred in the server ---

Message: null

Stacktrace:

java.lang.UnsupportedOperationException
	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerWrapper.generateLoadReport(ExtensibleLoadManagerWrapper.java:121)
	at org.apache.pulsar.broker.admin.impl.BrokerStatsBase.getLoadReport(BrokerStatsBase.java:156)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:359)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at org.apache.pulsar.broker.web.WebService$FilterInitializer$WaitUntilPulsarServiceIsReadyForIncomingRequestsFilter.doFilter(WebService.java:336)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlets.QoSFilter.doFilter(QoSFilter.java:202)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)

Broker logs are:

│ pulsar-broker     at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[org.eclipse.jetty-jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                                                   │
│ pulsar-broker     at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[org.eclipse.jetty-jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                                │
│ pulsar-broker     at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[org.eclipse.jetty-jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                                       │
│ pulsar-broker     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[org.eclipse.jetty-jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                                         │
│ pulsar-broker     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[org.eclipse.jetty-jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                               │
│ pulsar-broker     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[org.eclipse.jetty-jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                   │
│ pulsar-broker     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[org.eclipse.jetty-jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                                             │
│ pulsar-broker     at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[org.eclipse.jetty-jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                                          │
│ pulsar-broker     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[org.eclipse.jetty-jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                      │
│ pulsar-broker     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[org.eclipse.jetty-jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                    │
│ pulsar-broker     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[org.eclipse.jetty-jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                   │
│ pulsar-broker     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[org.eclipse.jetty-jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                                          │
│ pulsar-broker     at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[org.eclipse.jetty-jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]                                                                                                    │
│ pulsar-broker     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]                                                                                                                                                                                                │
│ pulsar-broker     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]                                                                                                                                                                                               │
│ pulsar-broker     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.119.Final.jar:4.1.119.Final]

Is there any config that would be missing? Thank you for your help and let me know if I need to open an issue on Github 🙂

Hi all, I'm having an issue with the log topics. In a nutshell, I'm not getting logs in the log topic for the first time the function runs after being updated. If the function runs a second time, logs for this new run show up in the topic. I have an

enricher

function with its log topic defined as persistent://public/default/enricher-log and I'm consuming from this topic. When I do a

functions update

operation, followed by producing a message, I don't get the logs from my application in the

<persistent://public/default/enricher-log>

log topic. Instead I get these Pulsar built-in logs:

----- got message -----
publishTime:[1744355218993], eventTime:[11651379494838], key:[null], properties:[], content:2025/04/11 07:06:52.094 log.go:47: [info] The default config file path is: conf/conf.yaml

----- got message -----
publishTime:[1744355218993], eventTime:[11651379494838], key:[null], properties:[], content:2025/04/11 07:06:52.100 asm_amd64.s:1700: [info] [Connecting to broker] remote_addr=<pulsar://pulsar-broker:6650>

----- got message -----
publishTime:[1744355218993], eventTime:[11651379494838], key:[null], properties:[], content:2025/04/11 07:06:52.102 asm_amd64.s:1700: [info] [TCP connection established] remote_addr=<pulsar://pulsar-broker:6650> local_addr=10.1.1.222:42440

----- got message -----
publishTime:[1744355218993], eventTime:[11651379494838], key:[null], properties:[], content:2025/04/11 07:06:52.105 asm_amd64.s:1700: [info] [Connection is ready] remote_addr=<pulsar://pulsar-broker:6650> local_addr=10.1.1.222:42440

...

I can see my application's logs only if I do another produce operation. They look like this:

----- got message -----
publishTime:[1744355308529], eventTime:[11651379494838], key:[null], properties:[], content:2025/04/11 07:06:58.993 logger.go:103: [info] got dimensions file="queuefront-data-pipeline/PulsarFunctions/handlers/enricher/enricher.go:56" functionName="PulsarFunctions/handlers/enricher.Enricher.Handle" partitionKey="" pulsarFunctionId="ad5e57a1-e12e-4010-b213-aabaeb6e67e9" pulsarFunctionName="qfdp-enricher" pulsarFunctionVersion="342c02e3-cdf8-4291-ad2a-e7439d2af1bf" traceId="f0a18b35-0be3-4a7b-97fd-b741c6445467" unusualRequestDimensions=[]qfdp_enriched_record.Dimension{}

----- got message -----
publishTime:[1744355308529], eventTime:[11651379494838], key:[null], properties:[], content:2025/04/11 07:06:58.993 logger.go:103: [info] got dimensions assetRequestDimensions=[]qfdp_enriched_record.Dimension{} file="queuefront-data-pipeline/PulsarFunctions/handlers/enricher/enricher.go:56" functionName="PulsarFunctions/handlers/enricher.Enricher.Handle" partitionKey="" pulsarFunctionId="ad5e57a1-e12e-4010-b213-aabaeb6e67e9" pulsarFunctionName="qfdp-enricher" pulsarFunctionVersion="342c02e3-cdf8-4291-ad2a-e7439d2af1bf" traceId="f0a18b35-0be3-4a7b-97fd-b741c6445467"

In the K8s pod of the function, I can see logs for both the runs. Any clue why the logs for the first time the function runs an update don't show up? Other details: • Running on my local K8s • Function is in Go and using the pulsar logger util: github.com/apache/pulsar/pulsar-function-go/logutil • Functions are running in clustermode.

Hi all, What is the process of submitting case-study on the platform pulsar-site. I have created a PR for adding such. Trying to make sure i am not missing anything. thanks

Hello all, I have pulsar standalone running in docker and I set up a jdbc sink connector for postgres but i get a connection error at 127.0.0.1:46301 and I don't know what might be causing this. Any help would be appreciated.

ERROR org.apache.pulsar.functions.runtime.process.ProcessRuntime - Health check failed for jdbc-postgres-sink-0
 java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
     at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
     at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
     at org.apache.pulsar.functions.runtime.process.ProcessRuntime.lambda$start$1(ProcessRuntime.java:187) ~[org.apache.pulsar-pulsar-functions-runtime-4.0.0.jar:4.0.0]
     at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:54) ~[org.apache.pulsar-pulsar-common-4.0.0.jar:4.0.0]
     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
     at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:?]
     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
     at java.base/java.lang.Thread.run(Unknown Source) [?:?]
 Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
     at io.grpc.Status.asRuntimeException(Status.java:539) ~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
     at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:548) ~[io.grpc-grpc-stub-1.56.1.jar:1.56.1]
     at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
     ... 1 more
 Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /127.0.0.1:46301
 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
     at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
     ... 1 more

i have this jdbc sink config

tenant: "public"
namespace: "default"
name: "jdbc-postgres-sink"
archive: "/pulsar/connectors/pulsar-io-jdbc-postgres-4.0.0.nar"  
className: "org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink"
inputs: 
  - "<persistent://public/default/temp>" 
configs:
  userName: "user1"
  password: "1234567890"
  jdbcUrl: "jdbc:<postgresql://postgres:5432/mqtt_db>"
  tableName: "test"
  key: "message_id"
  non_key: [ "temperature" , "timestamp" ]
  insertMode: UPSERT

hello all, im exploring deployments using helm charts, for version 4.0.1. The current documentation indicate "live upgrades", does this mean zero downtime? I'm trying to understand the longer term roadmap if zero downtime is not yet achieved. Thank you for your guidance and if there's another channel please provide.

Hi everyone, I have created two PR's on the

pulsar-site

• PR: Updating the article page with a blog post • PR: Fix tag rendering in Card component for resource entries A review would be greatly appreciated—thanks in advance!

I'm using pulsar-resource-operator on k8s. On the example here: https://github.com/streamnative/pulsar-resources-operator/blob/main/docs/pulsar_sink.md, there's a secret

SECRET1

, but there's no example how to reference

SECRET1

in the spec config. I tried this:

spec:
  configs:
    topic: $(MY_SECRET)
  secrets:
    MY_SECRET:
      key: MY_SECRET
      path: my-secret

but it fails with recognizing the value as a literal string

$(MY_SECRET)

. Is there a way to reference the k8s secret in the resource definition?

Hi! I am trying to install c++ pulsar client on Ubuntu machine. I have installed standalone pulsar. After installing the client I can see the libpulsar.so library installed in /usr/lib. I am trying to create a producer for which I am following this example. https://pulsar.apache.org/docs/4.0.x/client-libraries-cpp-use/ The example code given here needs Client.h file which is supposed to be in /usr/include/pulsar folder. But I dont see pulsar folder getting created anywhere and I dont find the header files installed anywhere in my environment. The documentation does not mention anything about installing these header files. How do I install these header files? I am new to c++. Do I need to run some cmake, make files? Your help is greatly appreciated.

Hi all, My name is Feng WenZhi, one of the Apache Pulsar Committer with Github id: thetumbled. I wrote a book in Chinese on how to maintain and optimize Pulsar service before, which is distributed to some participants at the Pulsar Meetup through an application process. Now I am going to rewrite these materials to make them more accessible and easier to understand as a brand new digital book, titled Practical Optimization of Apache Pulsar. Most importantly, i will open source this book in Github and release it in both english and Chinese. You can read it at GitBook: https://tumbleds-library.gitbook.io/thetumbleds-library. and star my Github repository to receive the latest update: https://github.com/thetumbled/Practical-Optimization-of-Apache-Pulsar. If you are interested in exploring more topics, feel free to leave a message to let me know, and I will consider incorporating new topics into the book.

Hello, Team I am experiencing an issue with Pulsar where old messages are being re-delivered, and I'm seeing "Recovering from bookkeeper ledger cursor" in the logs. It seems related to cursor recovery and acknowledgment handling. Any advice on diagnosing and fixing this would be greatly appreciated! I happens whenever broker restarts. Also, there is no issue in acknowledging the message. PS: I am using delay delivery functionality here. pulsar version 2.10.0 2025-04-18T184612,598+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/UPDATE_HOTEL_PER_BRAND] Recovering from bookkeeper ledger cursor: UPDATE_HOTEL_PER_BRAND-subscription 2025-04-18T184612,444+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/DISTRIBUTE_HOTEL_CURATION_TASKS] Loading cursor DISTRIBUTE_HOTEL_CURATION_TASKS-subscription 2025-04-18T184612,431+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/UPDATE_HOTEL_PER_BRAND] Loading cursor UPDATE_HOTEL_PER_BRAND-subscription

Hello everyone. I produce MQTT messages using a java producer that sends the payload in a byte[] array to the topic. Afterwards I get this error in my JDCB postgres sink connector

ERROR org.apache.pulsar.io.jdbc.JdbcAbstractSink - Got exception Primitive schema is not supported: BYTES after 1 ms, failing 1 messages
java.lang.UnsupportedOperationException: Primitive schema is not supported: BYTES

Is there a way to insert the messages to the db using the jdbc sink connector without transforming the byte[] array? I basically want to do a simple producer->pulsar->postgres performance test without transformations.

Hello! I am configuring tiered storage offloading for my pulsar cluster. I am looking to offload to AWS S3. Typically, we require server-side encryption when uploading to our S3 buckets. Is there a way to configure the offloader to use a KMS key for server-side encryption?

got error when upgrading to 3.0.11 ``Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.bookkeeper.mledger.ManagedLedgerException: Bookie operation timeout error code: -23\n\nStacktrace:\n\norg.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Bookie operation timeout error code: -23\n\tat org.apache.pulsar.broker.service.BrokerService$2.openLedgerFailed(BrokerService.java:1861)\n\tat org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$10(ManagedLedgerFactoryImpl.java:429)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(ManagedLedgerFactoryImpl.java:420)\n\tat org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$1.lambda$operationComplete$2(ManagedLedgerImpl.java:443)\n\tat org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137)\n\tat org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.apache.bookkeeper.mledger.ManagedLedgerException: Bookie operation timeout error code: -23\n"} it seems now kubenerbetes cluster has to do a full restart .

👋 Hello, we've been having some DirectMemory OOM issues running Pulsar (v3.2.1; but confirmed on v4.0.2 as well) that I was hoping some experts here might have some ideas on how to resolve. We have a ruby app that utilizes the REST API to publish messages to a topic with a rust-based consumer that consumes the topic. Almost daily, the

ownerBroker

of the topic will OOM with

2025-01-27T21:08:36,820+0000 [pulsar-io-7-5] ERROR org.apache.pulsar.PulsarBrokerStarter - -- Shutting down - Received OOM exception: failed to allocate 4194304 byte(s) of direct memory (used: 2147483648, max: 2147483648) io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 2147483648, max: 2147483648)

I'll add some other notes in thread.

Hi all, My name is Feng WenZhi, one of the Apache Pulsar Committer with Github id: thetumbled. The first part of my open source book, Practical Optimization of Apache Pulsar, is completed. This part focuses on the load - balancing algorithm. Comprising a total of 6 chapters, this part enables readers to grasp the principles and practical guidelines of the load - balancing algorithm. With this book, you will be able to master how to optimize Pulsar service for the best performance. Next part will be about the delay message queue, negative acknowledgment and retry/dead letter queue feature, both of which is related to the delayed message processing in Pulsar. You can access the content on GitBook at: https://tumbleds-library.gitbook.io/thetumbleds-library. Additionally, please star my GitHub repository at https://github.com/thetumbled/Practical-Optimization-of-Apache-Pulsar to stay updated with the latest information. If you are interested in exploring more topics, feel free to leave a message to let me know, and I will consider incorporating new topics into the book.

@Lari Hotari When will the below issue fixed ? https://github.com/apache/pulsar/issues/22939 Since jetty version used is nearing EOL.

Hello wondering if there are any pointers to run a kafka-connect cluster with pulsar adapters to use pulsar as the underlying transport.

Hi all, Is there any known bug where the incorrect remote endpoint protocol version gets picked up even when the protocol version is correct, for an established connection? We are facing one issue: while creating a system client for making a reader on the system topic(__change_events), it gets stuck on protocol version 0 while checking for more events. This issue was there for all the topics for a single broker. A few things to add here are that the issue started after the bundles reassignment happened on this broker and was mitigated after unloading the topic and deleting change_events, as it could have re-established the connections. Note: we are on pulsar version 3.0.7

I have a topic per user. Is it reasonable to let a user wait for topic updates with long-polling? The user will send an http request, the server will create a temporary reader, return existing messages or wait 30 seconds for updates and return. Will creating a reader every 30 seconds per topic a large overhead?

Does someone have any experience with deploying a pulsar topic with a JSON schema using the pulsar resources operator? It behaves differently than using the same JSON schema and uploading it via pulsar-admin. https://github.com/streamnative/pulsar-resources-operator/issues/311