pulsar

Hi Eric, if you’re seeing `availablePermits = -1`  it means the clients queue is full. You can check `unackedMessages` to confirm if the broker has dispatched messages that have not been ack’d by the consumer

thanks - that make sense, and i do see the `unackedMessages` but the throughput is zero and stays zero which is what throws me off. i do see that my function is still reading/writing, so obviously its receiving messages.

really good questions <@U05B8TPRCG6>. There should be better docs which show how to troubleshoot such problems.

btw. <https://pulsar.apache.org/docs/3.0.x/developing-binary-protocol/#consumer> contains some high-level explanation of the flow control where "permits" are used.

One detail is that the client view of the available permits isn't the same as the view on the broker. The broker view can be seen in the Pulsar stats, but the client's side view is harder to observe. I've usually used heap dumps and inspected the heap content with Eclipse Memory Analyzer (MAT) using <https://wiki.eclipse.org/MemoryAnalyzer/OQL|OQL> or <https://github.com/vlsi/mat-calcite-plugin|MAT Calcite SQL> to query the state such as permits.

Explaining the troubleshooting process would be a useful conference talk. In many cases, the problem is a client application issue. In rare cases, there are client or broker bugs. Which Pulsar client version &amp; language library? Which Pulsar broker version?

&gt; i do see that my function is still reading/writing, so obviously its receiving messages
That might be the function processing &amp; re-processing the client side queue.

interesting - i can tell there are actually messages being processing and sent downstream, because i can see the size of the elastic index i'm sinking to growing.

it's pulsar 2.10, and the functions are written in python

lari - i believe its 2.10.0, it's a managed service running on DataStax