https://pulsar.apache.org/ logo
Join Slack
Powered by
If I’m seeing producer timeouts, what things shoul...
# general
a
If I’m seeing producer timeouts, what things should I look at?
👀 1
if it helps, randomly this appears to occur it doesn’t seem constant
and I wouldn’t say it correlates with traffic being high either
s
Timeout can happen in many places. It can be at the client side limiting by 
maxPendingMessages
or can be at the server side (i.e., jvm pause, latency increase at the storage layer, etc).
a
I’ll look into that client side setting for sure, is there anything to be done to handle “jvm pauses” better? I think I can rule out the storage layer
s
You can checkout the JVM metrics to see if it is impacted by JVM
➕ 1
d
JVM metrics will help you determine if the heap is sized properly and if you are using the proper GC settings. Can you share the JVM configs you are using to run the client application?
l
which Pulsar client and which version? Is sendTimeout setting at the default value? (not set)
a
David
Copy code
pulsar:
  zookeeper:
    configData:
      PULSAR_MEM: >
        -Xms3g -Xmx3g

  broker:
    configData:
      PULSAR_MEM: >
        -Xms4g
        -Xmx4g
        -XX:MaxDirectMemorySize=19g

  proxy:
    configData:
      PULSAR_MEM: >
        -Xms2g
        -Xmx2g
        -XX:MaxDirectMemorySize=2g

  bookkeeper:
    configData:
      PULSAR_MEM: >
        -Xms4g
        -Xmx4g
        -XX:MaxDirectMemorySize=19g
and Lari the C++ client, appears to be on the latest version, and we’re using the 
3.1.2
pulsar chart for the cluster. I don’t see the 
sendTimeout
being set anywhere
I’m working on throwing a Java Grafana dashboard together - it’s just not a language I’m skilled at
👍 1
l
and Lari the C++ client, appears to be on the latest version, and we’re using the 
3.1.2
pulsar chart for the cluster. I don’t see the 
sendTimeout
being set anywhere
do you mean Pulsar version 
3.1.2
? I'd recommend switching to 
3.0.4
or 
3.2.2
since `3.1.x`isn't maintained (supported versions) .
a
gotcha, I can definitely upgrade it’s just a bit worrying while it’s seemingly unstable
l
getting a producer timeout could mean that there's a bug that is causing things to get blocked on the broker side. It could also be a performance issue where the broker & bookies cannot keep up. Usually there would be other metrics to find that out.
there are also a few known issues in 3.0.4 / 3.2.2 so it's not perfect. Hopefully we'd get 3.0.5 & 3.2.3 released soon.
for diagnostics in a blocked broker situation, you could collect thread dumps and heap dumps. One script for being able to do this over kubectl is https://github.com/lhotari/pulsar-contributor-toolbox/blob/master/scripts/collect_jvm_diagnostics_from_pod.sh (and https://github.com/lhotari/pulsar-contributor-toolbox/blob/master/scripts/collect_jvm_diagnostics_from_pods.sh) . thread dumps don't usually contain sensitive info (other than zookeeper host names), but heap dumps contain the memory content so that's something that shouldn't be ever shared publicly.
m
Alex, can you also share the resources allocated on cpu/memory requests/limits. https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1715177677956329?thread_ts=1715111220.906309&cid=C5Z4T36F7
5 Views
Open in Slack
PreviousNext
Powered by