Hi, has anyone encountered a transaction pending-a...
# general
d
Hi, has anyone encountered a transaction pending-ack replay thread getting stuck indefinitely? On Pulsar 4.0.10, I observed a pulsar-transaction-executor-N-1 thread stuck in MLPendingAckStore$PendingAckReplay.run(), repeatedly calling Thread.sleep(1). Because these executors are single-threaded and shared by multiple subscriptions, one stuck replay can prevent consumers from being created on unrelated subscriptions. I also observed broker pod restarts around the same time. I suspect the stuck replay may block the health-check subscription when it hashes to the affected executor, causing Kubernetes health checks to fail and restart the broker pod. My current theory is: - A managed-ledger read is issued, but its callback never arrives. - outstandingReadsRequests remains 1, the entry queue stays empty, and isReadable remains true. - PendingAckReplay therefore loops forever. - Normally, ManagedLedgerImpl.checkReadTimeout() should terminate the read, but this protection is disabled when managedLedgerReadEntryTimeoutSeconds=0. - A possible trigger is an empty or newly rolled-over pending-ack ledger being deleted while replay is reading it, because new ledgers initially have timestamp=0 and system ledgers use retention 0. I found related issues #14852, #23506, and #24401, but none appears to fully address pending-ack replay when the read callback never returns. Has anyone seen this behavior? Does this analysis seem plausible? Could this explain the broker pod restarts, and is there a recommended configuration to avoid it besides setting managedLedgerReadEntryTimeoutSeconds to a non-zero value? I have a repeatable local reproduction and thread dumps that I’m happy to share.
l
please report a GitHub issue at github.com/apache/pulsar/…/choose
d
👍 1