This message was deleted.
# generals
Slackbot
09/01/2023, 10:15 AMThis message was deleted.
y
Yu Wei Sung
09/01/2023, 1:24 PMdo you have topic stats?
d
David K
09/01/2023, 3:18 PMI know it is a bit of a cliche, but given that you are running a really old version of Pulsar, you might want to consider upgrading to 2.10.x +
David K
09/01/2023, 3:20 PMAlso, please share topic-stats-internal. When I have seen this situation in the past, it is often the case where there are some gaps in the message acknowledgements. This results in the ledgers with the missing acks being retained, which leads to a large storage size and backlog size
đź‘Ť 1
n
Nikhil O
09/02/2023, 1:54 AMAfter much fiddling I found out that the topics with large storage sizes had one or two messages from 2022. And I’m guessing the consumers were not able to process/ack them. So after skipping those messages the backlog storage size went from 150GB to 16GB.
Nikhil O
09/02/2023, 1:55 AMHow did 1 or 2 messages inflate the backlog storage size like this ? Does backlog storage size increase with failed acknowledgments or retries ?
d
David K
09/02/2023, 2:12 AMWhen a message isn't acknowledged, then the entire ledger has to be retained. A ledger contains 50k messages, which can be many MB in size.
David K
09/02/2023, 2:13 AMIf that happens across multiple ledgers/messages then the backlog size can get quiet largw
n
Nikhil O
09/04/2023, 5:44 AMI would have to understand how a message is stored in the ledger. It was just one message.
Does this mean that if a message from 2022 isn't acknowledged, then all the messages since that message are retained until that one is acknowledge/skipped ?
d
David K
09/05/2023, 2:26 PMBy default, each ledger is allowed to grow to ~50k entries before it is closed, and put into read-only mode. If you have batching enabled on the producer, than each “entry” contains not one, but several messages. Therefore the potential size in terms of message count (backlog), and physical storage can be very big. Now, in a missing ack scenario where one of these entries were delivered but never ack’ed, the entire ledger itself MUST be retained. This is because it is an append-only log. So even a single unack’ed message could result in a backlog of a ~500k messages (with an average batching size of 10), and a storage size of 5-6GB depending on the message size. This effect can easily compound if there are multiple ledgers with unack’ed messages.
2 Views