Hi Team 👋 I'm investigating an issue related to the Pulsar broker configuration parameter

delayedDeliveryTrackerFactoryClassName

. We initially used

InMemoryDelayedDeliveryTracker

, (because we where using version 2.7 of pulsar) which caused acknowledged delayed messages to be reprocessed after a broker restart likely due to its state stored only in memory. Given our high message volume (millions), this behavior is problematic. A screenshot is available showing the lag escalation following a broker restart. We're generating delayed messages out of sequence, resulting in gaps within the acknowledged message stream. This causes non-contiguous ranges of messages to be marked as deleted or eligible for deletion. In our screenshot, the value of

nonContiguousDeletedMessagesRanges

is 16833. To mitigate this following the upgrade of pulsar to the version 4.0.4, we updated the broker config to use

org.apache.pulsar.broker.delayed.BucketDelayedDeliveryTrackerFactory

, which should persist delayed delivery metadata to disk via BookKeeper ledger buckets. However, after switching to the bucket-based tracker, we're still seeing the same behavior post-restart. A few observations and questions: • I checked the

pulsar_delayed_message_index_loaded

metric and noticed that messages are still being loaded into memory, while

pulsar_delayed_message_index_bucket_total

remains at zero. Is this expected? Shouldn’t the bucket tracker be persisting and loading from disk? • Are there additional broker settings required to fully enable bucket-based delayed delivery tracking? For example: ◦ Do we need to explicitly configure

delayedDeliveryTrackerBucketSize

or

delayedDeliveryMaxNumBuckets

? ◦ Is there any dependency on topic-level settings or namespace policies that could override the broker-level tracker configuration? ◦ Could other settings interfere with delayed message persistence? Any insights or guidance would be greatly appreciated. Thanks for your help!

there are more details in https://github.com/apache/pulsar/discussions/23990 . It doesn't contain an explicit list of settings which you should be applying, but I expect that you'd be able to make progress. There's also https://github.com/apache/pulsar/pull/24392 and https://github.com/apache/pulsar/pull/24391 which are part of Pulsar 4.0.6 which added the missing settings to broker.conf. I'd recommend to use the latest Pulsar 4.0.x version, 4.0.7 instead of using Pulsar 4.0.4.

Thank you for your answer. We will try that today.

> Do you have any recommendations for setting the values of

managedLedgerMaxUnackedRangesToPersist

and

managedLedgerMaxBatchDeletedIndexToPersist

? > Should we expect a significant deviation from the 830K? Around 1M? Unfortunately, I haven't tested with different variations that high. PR 9292 introduced

managedLedgerPersistIndividualAckAsLongArray=true

format https://github.com/apache/pulsar/pull/9292 and it says > Serializing OpenRangeSet can allow managed-cursor to store 10M ranges with 5MB data size. Although the PR says this, I don't think that a test exists to validate this. Therefore it's better to test that it can really do this. One way to test would be to write a unit test for a worst case payload for "managed cursor info" and ensuring that the compressed size doesn't exceed Bookkeeper server's

nettyMaxFrameSizeBytes

(default value is 5MB) and broker's

bookkeeper_nettyMaxFrameSizeBytes

(defaults to

maxMessageSize

+ 10240) . (It's possible to increase

nettyMaxFrameSizeBytes

in bookkeeper.conf and

bookkeeper_nettyMaxFrameSizeBytes

in broker.conf to adapt to larger payloads.) You could also test in practice, but a unit test could be easier to control and simulate a worst case situation. Here's one unit test https://github.com/apache/pulsar/blob/master/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorInfoMetadataTest.java . However I think it's creating the ManagedCursorInfo in the

managedLedgerPersistIndividualAckAsLongArray=false

format, so it would require adapting it to encode the "unacked ranges" in the

managedLedgerPersistIndividualAckAsLongArray=true

format.

Hello @Lari Hotari, Thanks again for your response. I’m not very familiar with the Pulsar codebase so I don't know if I can really make this unit test. We’ll aim to replicate the same value in the DEV environment for testing. Thank you again!

Testing with a worse case scenario in dev/staging should be a sufficient validation while testing that the full state is recovered after topic unloading and loading.