So, building off the previous, I tried to stress test our setup a little. The basic idea was to create a thousand topics, have a producer for each, write a thousand messages to each topic and read them back. The serial one ran for over two hours before the OOM killer killed it. So, I tried parallelizing it, ten producer workers at a time, each writing to their own topic. It stalled on the writes for a while and then said:

WARNING:pulsar:[10.20.3.78:34944 -> 10.20.2.147:6650] Received send error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6
WARNING:pulsar:[10.20.3.78:34960 -> 10.20.2.147:6650] Received send error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6

over and over until I interrupted it. There are four bookies in this cluster. Thoughts welcome

If you happen to be using Apache Pulsar Helm chart for the deployment, it's necessary to increase

PULSAR_MEM

for all Pulsar components. https://github.com/apache/pulsar-helm-chart/blob/c341e7d65d8d351adc1f866acc2bf67c51a96b5c/charts/pulsar/values.yaml#L842-L845 Unfortunately, nobody has ever contributed a guide for tuning Apache Pulsar Helm chart for actual production usage. That would be a useful contribution for new users of Pulsar. The current documentation only touches upon the subject: https://pulsar.apache.org/docs/4.1.x/helm-deploy/#cpu-and-ram-resource-requirements

Thanks, will try this this morning. 0