Hi, has anyone encountered a transaction pending-ack replay thread getting stuck indefinitely?
On Pulsar 4.0.10, I observed a pulsar-transaction-executor-N-1 thread stuck in MLPendingAckStore$PendingAckReplay.run(), repeatedly calling Thread.sleep(1). Because these executors are single-threaded and shared by multiple subscriptions, one stuck replay can prevent consumers from being created on unrelated subscriptions.
I also observed broker pod restarts around the same time. I suspect the stuck replay may block the health-check subscription when it hashes to the affected executor, causing Kubernetes health checks to fail and restart the broker pod.
My current theory is:
- A managed-ledger read is issued, but its callback never arrives.
- outstandingReadsRequests remains 1, the entry queue stays empty, and isReadable remains true.
- PendingAckReplay therefore loops forever.
- Normally, ManagedLedgerImpl.checkReadTimeout() should terminate the read, but this protection is disabled when managedLedgerReadEntryTimeoutSeconds=0.
- A possible trigger is an empty or newly rolled-over pending-ack ledger being deleted while replay is reading it, because new ledgers initially have timestamp=0 and system ledgers use retention 0.
I found related issues #14852, #23506, and #24401, but none appears to fully address pending-ack replay when the read callback never returns.
Has anyone seen this behavior? Does this analysis seem plausible? Could this explain the broker pod restarts, and is there a recommended configuration to avoid it besides setting managedLedgerReadEntryTimeoutSeconds to a non-zero value?
I have a repeatable local reproduction and thread dumps that I’m happy to share.