This message was deleted.
# general
s
This message was deleted.
p
Do you have delayed messages in the topic?
s
We had initially, but recently turned it off. Still seeing consumers getting stuck.
p
Ok, I think you said turned it off means you will not publish new delayed messages to the topic, but maybe there are some old delayed messages still in the topic, right?
s
Yes, possible.
p
Or all the delayed messages are cleaned up
s
There might still be some delayed messages in the backlog.
p
It looks like related to this fix https://github.com/apache/pulsar/pull/20233, but only released in 2.10.5 or later version.
s
One question. Does this behaviour occur both cases when we have producer side
delveryAfter
and/or consumer side
negativeAckRedeliveryDelay
?
p
No, it only occur when the producer publish messages with
delveryAfter
s
understood. So looking at the description of the fix , it seems like it is happening only in case of strict ordering (i.e.
allowOutOfOrderDelivery=false
) , Am i correct here?
p
No, I have tried to enable
allowOutOfOrderDelivery
for the test that was added by the PR, but without the fix. The test still fails.
s
Thanks 🙂 . This fix is available in some minor version in 2.9? or from 2.10.5?
p
It is released in 2.10.5
s
So i verified all the messages coming in are published after i turned off the
deliveryAfter
. But still now I see many consumers in the pipeline are sitting idle (while only 1 or 2 consumers actually polling and processing). And interestingly , when i do negative acknowledgement for a particular message, its not coming back for re-processing within default negative ack redelivery delay (which is 1 sec).
My use-case is quite simple: I want to achieve Pulsar as a proper queue. Process individual keys in order Process. If there is a validation failure during processing, negatively ack the message to re-process it again later. I am using Key_shared subscription to achieve the same. But somehow message dispatch from broker to consumers seems flaky. Even if there is backlog, often most of the times except for 1-2 consumers rest remain idle. Restart works. See more even distribution of polled messages across consumers for some time. But it comes back in the same state within few hours. Before pipeline goes to a stuck state, last logs are always some consumer doing negative ack. Not sure how I am using negative ack is wrong or not. I am not using retry-letter topic. Relying upon the negatively acked messages to get redelivered by the broker to existing consumers. I am negatively ack upto 10 times, then handling it in the code to ack when this number is exhausted.
p
Could you please share your topic's stats and internal-stats? They will provide some information that can help understand why consumers stop processing messages.