This message was deleted.

there are around 7 bookies, 10 brokers serving 400KBps publish, 120MBps dispatch workload

It might happen if the topic owner moves to another broker (load balance). The new owner broker will load the topic because the new owner broker does not have an entry cache for this topic, so the broker will read from the bookies directly to deliver messages to consumers. • Option 1: reduce the max read batch to bookies (broker.conf)

# Max number of entries to read from bookkeeper. By default it is 100 entries.
dispatcherMaxReadBatchSize=100

# Max size in bytes of entries to read from bookkeeper. By default it is 5MB.
dispatcherMaxReadSizeBytes=5242880

You can try to decrease the read batch size to reduce the requests to the bookie side. This is a known part we need to improve. The bookie API doesn't support batch reading, so the broker will send 100 read requests for each subscription. So, at most, you will have 100 * 300 read requests to bookies for a topic. BTW, the improvement has already started on the bookkeeper side. After the batch read API is available, then we can integrate Pulsar to adapt the new batch read API. • Option 2: disable the sticky read (broker.conf)

# Enable/disable having read operations for a ledger to be sticky to a single bookie.
# If this flag is enabled, the client will use one single bookie (by preference) to read
# all entries for a ledger.
# Please notice that enabling sticky reads also requires that the ensemble size equals write quorum size for the ledger's
# persistence policies.
bookkeeperEnableStickyReads=true

Sticky Read is a performance optimization for data reading by leveraging the bookie's read-ahead cache. But the downside is all the read requests on a ledger will go to the same bookie. For your case, disabling the bookkeeper sticky reads will help spread out the requests to multiple replicas. • Option 3: Increase the bookie pending read (bookkeeper.conf)

# Number of threads that should handle read requests. if zero, the reads would
# be handled by netty threads directly.
numReadWorkerThreads=8

# If read workers threads are enabled, limit the number of pending requests, to
# avoid the executor queue to grow indefinitely
maxPendingReadRequestsPerThread=2500

By default, one bookie will have 2500 * 8 max pending reads. If your bookie has more memory available, you can try to increase the pending requests.

Thanks for having a look! Actually, since this is almost a constant number in the metrics for over many hours now, I am assuming it is not due to load balancing of topics. Option 1 sounds good, we will try it out. For option 2, since almost all the bookies are having this issue, will it make a difference? I checked the property mentioned in option 3 and it seems like the default value is 10000? We seem to be relying on the default value there.

For option 2, if you only a few topics with 300 subs, it should work, if you have many topics with 300 subs, it will not work. For option3, you’d better check your configuration file. I just copied from the master branch of pulsar.

Yes, there is only one topic with 300 subs. But disabling sticky means the calls would be more spread out, but the error is coming from all brokers for all bookies, will making the call spread to replicas make a difference?

It will make a difference. With sticky read enabled, for one ledger, all the read requests will go to the same bookie. If you disable it, the read requests can go to N bookies. So each bookie will get less read request.

But eventually if there are 10000 read requests in a second and 10 bookies, then in either situation, won't it be 1000 read request per second per bookie?

If the 10000 reads happened on the same ledger. They are different sticky read enabled - one bookie get 10000 reads requests sticky read disabled - one bookie might get 3000-4000 requests if your replica is 3

I see. So this is more of point in time rather than averaging out over a second? We have 30 partitions, so 30 ledgers open at any time, so the spread of those 30 would already be over all 10 bookies..

Yes, reads for 30 ledgers should be spread out to many bookies.