@RunLLM producer stats show high latency as 3000 ms however the pulsar_storage_write_latency is less than 5ms this shows broker is taking time what could be the reason ?

If your producer stats show high latency (e.g., 3000 ms) but the

pulsar_storage_write_latency

metric is less than 5 ms, this indicates that the bottleneck is not in the storage layer (BookKeeper writes), but likely elsewhere—such as network delays, broker-side queuing, batching, or client-side issues. The

pulsar_storage_write_latency

metric specifically measures the time to persist entries to storage, not the total end-to-end publish latency experienced by the producer. High producer latency with low storage latency suggests the delay is before the message reaches storage, possibly due to broker overload, batching settings, or network issues, as described in StreamNative Cloud Metrics and Pulsar Prometheus Stats. Would you like more detail on how to diagnose or address this discrepancy? Sources: - StreamNative Cloud Metrics - Pulsar Prometheus Stats 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yes

To diagnose high producer latency when

pulsar_storage_write_latency

is low, follow these steps: 1. Check Broker Load and Queuing: High broker CPU, memory, or thread pool saturation can cause queuing delays before messages are written to storage. Review broker metrics like

pulsar_rate_in

,

pulsar_rate_out

, and thread pool usage. If these are high, consider scaling out brokers or tuning thread pools (StreamNative Cloud Metrics). 2. Review Batching and Producer Settings: If batching is enabled, large

batchingMaxPublishDelay

or high

batchingMaxMessages

can increase producer-side latency. For latency-sensitive workloads, set

batchingMaxPublishDelay

to a low value (e.g., 1 ms) or disable batching (Pulsar Perf Docs). 3. Network Latency: Network issues between producer and broker can add significant delay. Check for packet loss, high RTT, or unstable connections. 4. Backpressure and Flow Control: If the broker or topic is throttled (due to publish rate limits or backlog quotas), producers may be blocked. Check for rate limiting metrics and backlog status (StreamNative Cloud Metrics). 5. Client-Side Issues: If the producer is using synchronous sends or has a small

maxPendingMessages

, it may block waiting for acknowledgments. Increase

maxPendingMessages

or use async sends for higher throughput (Pulsar Client API). 6. Topic Unload or Failover: If the topic is being unloaded or the broker is failing over, producers may experience temporary unavailability, causing spikes in latency (Pulsar Client Go PR #81). Example: If you see high producer latency but

pulsar_storage_write_latency_le_5

is high (most writes <5ms), check the broker's CPU and thread pool metrics, reduce batching delay, and ensure the network is healthy. Would you like a code/config example for tuning producer settings? Sources: - StreamNative Cloud Metrics - Pulsar Perf Docs - Pulsar Client API - Pulsar Client Go PR #81 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM metrics for broker thread pool usage

Pulsar exposes broker thread pool usage metrics, including the number of threads and queue sizes for various executors (e.g., Netty IO, HTTP server, orderedExecutor). These are available as Prometheus metrics and can be monitored via Grafana dashboards. Key metrics include thread counts and queue sizes for IO, HTTP, and ordered executors (Pulsar Metrics Reference, StreamNative Monitor Docs). Would you like a list of specific metric names or details on how to visualize them? Sources: - Pulsar Metrics Reference - StreamNative Monitor Docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM Netty IO, HTTP server, orderedExecutor share the prometheus metrics name

It seems likely that Netty IO, HTTP server, and orderedExecutor each have their own specific Prometheus metrics and do not share the same metric names. For example, HTTP server thread pool metrics are exposed as

pulsar_web_executor_*

(such as

pulsar_web_executor_max_threads

,

pulsar_web_executor_active_threads

), while Netty IO and orderedExecutor metrics are tracked separately and are not grouped under a shared metric name (Pulsar Metrics Reference, PR #14742). Would you like a detailed mapping of metric names for each thread pool type? Sources: - PR #14742 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM pulsar-io-4-1" #108 prio=5 os_prio=0 cpu=4317956.40ms elapsed=671330.05s tid=0x00007efbb8002b90 nid=0x1fe runnable [0x00007efb26ef3000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:182) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:312) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:376) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@17.0.13/Thread.java:840) Is the IO doing good?

Yes, this thread is in a RUNNABLE state and is blocked in

epollWait

, which is normal for Netty IO threads waiting for network events. High CPU time and long elapsed time are expected for long-lived IO threads; this does not indicate a problem by itself (Pulsar Issue #23187, Pulsar PR #17870). Would you like more detail on interpreting Netty IO thread states or troubleshooting IO bottlenecks? Sources: - Pulsar Issue #23187 - Pulsar PR #17870 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM give me the tuning for broker where producer latency is high .

To tune a Pulsar broker for high producer latency, focus on increasing thread pool sizes, optimizing cache settings, and adjusting batching and rate limits. Key settings to consider are: - Increase

numIOThreads

,

numHttpServerThreads

, and

numOrderedExecutorThreads

to better handle concurrent requests (ServiceConfiguration.java). - Tune broker cache: set

maxMessagePublishBufferSizeInMB=500

,

managedLedgerCacheSizeMB=512

, and related cache eviction parameters (Pulsar Issue #23466). - Monitor and adjust publish rate limits (

brokerPublisherThrottlingMaxMessageRate

,

brokerPublisherThrottlingMaxByteRate

) if enabled (TopicPublishThrottlingInitTest.java). - Ensure network and disk are not bottlenecks, and consider scaling out brokers if resource usage is high. Would you like a sample configuration or more detail on each parameter? Sources: - ServiceConfiguration.java - Pulsar Issue #23466 - TopicPublishThrottlingInitTest.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM more details on each parameter

It seems likely that the following parameters are most relevant for tuning broker performance when producer latency is high: 1. numIOThreads: Controls the number of Netty IO threads for handling network connections. Increasing this allows the broker to process more concurrent network events, which can help if the broker is network-bound (ServiceConfiguration.java). 2. numHttpServerThreads: Sets the maximum number of threads for the HTTP server (Admin API and REST endpoints). If you see slow admin operations or REST calls, increasing this can help (ServiceConfiguration.java). 3. numOrderedExecutorThreads: Determines the number of threads for ordered executor pools, which handle ordered tasks such as message dispatching. More threads can reduce queuing delays for these operations (ServiceConfiguration.java). 4. maxMessagePublishBufferSizeInMB: Limits the total memory used for buffering messages before they are written to BookKeeper. Setting this to a reasonable value (e.g., 500 MB) prevents OOM errors and excessive GC, but if set too low, it can throttle producers (Pulsar Issue #23466). 5. managedLedgerCacheSizeMB: Sets the size of the managed ledger cache. A larger cache can improve read performance and reduce disk IO, but uses more memory (Pulsar Issue #23466). 6. brokerPublisherThrottlingMaxMessageRate / brokerPublisherThrottlingMaxByteRate: These parameters limit the publish rate at the broker level. If set too low, they can throttle producers and increase latency. Set to 0 to disable throttling (ServiceConfiguration.java). 7. dispatcherMaxReadBatchSize: Controls how many entries are read from BookKeeper in a single batch. Increasing this can improve throughput but may increase memory usage (ServiceConfiguration.java). 8. dispatcherMaxReadSizeBytes: Sets the maximum size in bytes for a single read batch from BookKeeper. Tune this to balance throughput and memory usage (ServiceConfiguration.java).

@RunLLM pulsar_broker_publish_latency_count{cluster="itomdipulsar"} 7564547.0 pulsar_broker_publish_latency_sum{cluster="itomdipulsar"} 3.3593571E7

The metric

pulsar_broker_publish_latency_count

shows the total number of publish operations (7,564,547), and

pulsar_broker_publish_latency_sum

is the cumulative publish latency in milliseconds (33,593,571 ms) for the cluster "itomdipulsar". To get the average publish latency, divide the sum by the count: 33,593,571 ms / 7,564,547 ≈ 4.44 ms per publish (StreamNative Cloud Metrics). Would you like help interpreting this further or comparing it to other latency metrics? Sources: - StreamNative Cloud Metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM pulsar_broker_publish_latency_sum{cluster="itomdipulsar"} 9526506.0 is good

The metric

pulsar_lb_memory_usage

reports the broker process memory usage as a percentage, used by Pulsar's load balancer to assess broker load. It is labeled by cluster and broker, and helps determine if a broker is overloaded or underutilized (Pulsar Metrics Reference). Would you like more detail on how this metric is calculated or how to use it for tuning? Sources: - Pulsar Metrics Reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM process_cpu_seconds_total

The metric

process_cpu_seconds_total

is a Prometheus counter that reports the total user and system CPU time consumed by the Pulsar process, measured in seconds. It helps monitor overall CPU usage of the broker or function process (StreamNative Cloud Metrics, Pulsar Metrics Reference). Would you like to know how to interpret this metric or use it for alerting? Sources: - StreamNative Cloud Metrics - Pulsar Metrics Reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM publish rate pulsar admin command to get and set

To get the publish rate for a topic, use:

pulsar-admin topics get-publish-rate <topic>

To set the publish rate for a topic, use:

pulsar-admin topics set-publish-rate <topic> --msg-publish-rate <messages/sec> --byte-publish-rate <bytes/sec>

(Pulsar Admin Reference, StreamNative pulsarctl get-publish-rate, StreamNative pulsarctl set-publish-rate) Would you like examples for namespace-level publish rate commands? Sources: - StreamNative pulsarctl get-publish-rate - StreamNative pulsarctl set-publish-rate 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM Give me the command to set publish rate to -1

To set the publish rate to -1 (which disables the publish rate limit) for a topic, use:

pulsar-admin topics set-publish-rate <topic> --msg-publish-rate -1 --byte-publish-rate -1

(StreamNative pulsarctl set-publish-rate) Would you like the equivalent command for namespaces or more details? Sources: - StreamNative pulsarctl set-publish-rate 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.