Hey, I had a question about increasing throughput for a consumer. Here are some details: • We run our consumers in kubernetes, with one consumer per pod • We use the node client, with the message

listener

pattern. Messages do not need to be processed in order • Some additional configurations: ◦ To keep debugging and DLQ management easier, we do not use

batching

◦ We use the

shared

subscriptionType ◦ the default

receiverQueue

size We've noticed that for a given consumer, all of the messages are being consumed in sequence. We'd like to increase concurrency on our nodes. Is the preferred approach to run multiple consumers on a node? or to tweak the listener to use a concurrency limit e.g.

p-limit

. Any advice appreciated 🙂

If you want to use multiple threads to process the messages and still use the listener, the easiest way is probably to: 1. Set N listener threads in the PulsarClient configuration 2. Create N consumers on the same subscription per each pod This will ensure all the N threads will be used to process messages

https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1710871462931239 -- this thread might help you

Thank you both 🙂 • @merlimat What would you consider best practice? we're not married to using the listener pattern • @Daniel did you land on creating multiple consumers in a pod? I created this discussion thread here if this could help others

Not yet! This is in our pipepline, though. And we plan to implement this by increasing the listenerThread count and creating multiple shared subscription consumers within the kube pod.

Ok, for now we opted to run 1 consumer and switching over to the

receive()

pattern