This message was deleted.
# general
s
This message was deleted.
a
You’re missing the “reason” part of the exception
Check the logs and I’ll help you with that - I know that code quite well
m
after restarting the broker sts I don’t see this error anymore, thanks @Asaf Mesika
But now i’m seeing some other interesting issue, from one of our apps that is publishing to pulsar I see these errors:
"Pulsar.Client.Api.LookupException: {\"errorMsg\":\"Failed due to too many pending lookup requests\"
a
In the broker, there is a in-flight protection mechanism, to avoid having too many concurrent requests for lookups
Copy code
@FieldContext(
        dynamic = true,
        category = CATEGORY_SERVER,
        doc = "Max number of concurrent lookup request broker allows to throttle heavy incoming lookup traffic")
    private int maxConcurrentLookupRequest = 50000;
Have you configured it differently?
Are you using reg-exp based subscriptions perhars?
m
We didn’t touch the in-flight protection
a
See my 2nd question 🆙
l
@Moty You seem to be hitting some bug that might be fixed in more recent versions of Pulsar. There are 214 commits between 2.11.0 and 2.11.1 . One possible cause is a deadlock in the broker. When you encounter an issue, a thread dump from each broker could help reveal deadlocks. If you run Pulsar with Kubernetes, this is a sample script that collects diagnostics information from a Pulsar pod: https://github.com/lhotari/pulsar-contributor-toolbox/blob/master/scripts/collect_jvm_diagnostics_from_pod.sh It also retrieves a heap dump (which contains sensitive information so don't share that publicly).
👍 1
@Moty do you have a chance to upgrade to 2.11.1 and see if the problem reproduces?
m
hmmm… I see.. then we will upgrade our cluster and see if it solves our issue
👍 2