Hello, We are experiencing an issue on some topics (persistent) in our cluster in production where some messages are consumed with a big delay after they are produced (45h or more). It seems that when the issue happens, it's located on specific partition(s) for a given topic. I think (maybe) that this issue occurs on topics that don't have much load, we're talking about a few messages a day because the heavy used topic don't seem to show that behavior, but can't be too sure as of right now. We also have more consumers (Pod/instance) for the "big" topics and they are all using a

shared

subscription with

16

partitions. We are using Pulsar

4.0.6

(via helm chart), our applications are using the Go client (commit

ed7d4980034871e1db28770576151c4c05c7d0ea

) I noticed this behavior a few times already. I know that it happened once when the brokers were restarted (rollout used to move workload to other k8s nodes, no config change there), I'm not sure if that's the case when applications (Go apps) restart and I'm still trying to see if this is a server or client problem. Is there anything that could potentially trigger that behavior? I'm thinking maybe partition discovery in the Go client, or an issue related to the partition consumer as I this there is an issue on Git: https://github.com/apache/pulsar-client-go/issues/1426 I'm not seeing anything obvious that could fix the issue in the release logs of

4.0.7

or

4.0.8

, or similar issues open on Git. Regardless, any feedback is welcome. Thanks for the help

Not sure if this is related, but one gap in Pulsar that could lead to consumers getting stuck is explained in https://github.com/apache/pulsar/issues/24926 . Besides OOM for Direct memory, it could also cause consumers to get stuck in a different case where OOM doesn't occur. This issue is not directly related to the client implementation. It would more likely occur when the client connection has very high volume topics and low volume topics sharing the same TCP/IP connection. In your case, were the pending messages delivered after new messages arrived to the topic?

I checked and it does not seem like pending messages are delivered after new messages arrive.

Is happening always with the same application and topic? Can you take some stats and internal stats every 3 minutes to see the server side changes once the issue arises?

• Do you think this is an actual issue? https://github.com/apache/pulsar-client-go/issues/1426

That's possible.

• Do you know of there is a place for this issue in

4.0.9

? https://github.com/apache/pulsar/issues/24926

It would be useful to address it sooner or later. The issue has been present forever, so it's not a new problem. It's a rare problem for most Pulsar users.

Thank you. In regards to stats, I will look at it next week and circle back here 🫡