@RunLLM On one of the cluster the org.apache.pulsar.client.impl.ProducerStatsRecorderImpl shows high latency Latency: med: 7002.000 ms - 95pct: 10907.000 ms - 99pct: 10996.000 ms - 99.9pct: 11020.000 ms - max: 11163.000 ms

High producer latency (e.g., median 7002 ms, 99th percentile 10996 ms) in org.apache.pulsar.client.impl.ProducerStatsRecorderImpl typically indicates broker or storage bottlenecks, network issues, or resource contention. Common causes include slow BookKeeper/bookie responses, broker IO thread saturation, high GC pauses, or excessive logging due to repeated exceptions, especially with unreachable clusters or replication failures. For example, excessive exception logging and thread contention can cause high publish latency and GC pauses, as seen in recent Pulsar issues and thread dumps (see details). To diagnose, check broker and bookie logs for errors, monitor broker IO/CPU/GC, and review network health. Also, ensure no misconfigured replication or unreachable clusters are causing repeated failures. Upgrading to the latest supported Pulsar version may resolve known performance bugs (see here). Would you like more detail on specific troubleshooting steps or metrics to monitor? Sources: - Bug: Broker seeing high latency, high CPU and high GC Pauses - Bug: Error while producing messages 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

producer response time is high

High producer response time is usually caused by broker or bookie (storage) slowness, network latency, or resource contention (CPU, IO, GC). Common issues include slow BookKeeper writes, broker IO thread saturation, or excessive logging due to repeated errors (see here). Also, misconfigured replication or unreachable clusters can cause repeated failures and high latency. Check broker and bookie logs for errors, monitor system resource usage, and ensure network health. Upgrading to a newer Pulsar version may help if you are on an older release (see here). Would you like more detail on how to troubleshoot or specific metrics to check? Sources: - Bug: Broker seeing high latency, high CPU and high GC Pauses - see here 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM broker io thread saturation

Broker IO thread saturation occurs when Pulsar's IO threads are fully utilized, causing delays in processing client requests and increasing producer/consumer latency. This can be triggered by high publish rates, slow BookKeeper responses, or excessive logging from repeated exceptions (see example). When IO threads are saturated, message processing slows down, leading to backpressure and timeouts. To address this, monitor IO thread usage, reduce publish rates, optimize BookKeeper performance, and minimize excessive logging. Consider increasing the number of IO threads if appropriate for your workload (see config discussion). Would you like more detail on monitoring or tuning IO threads? Sources: - Bug: Broker seeing high latency, high CPU and high GC Pauses - Able to use separated IO threads for BookKeeper Client 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM monitor IO thread usage, in broker

To monitor IO thread usage in a Pulsar broker, you can use Prometheus metrics exposed at the broker's /metrics endpoint, which include thread and CPU usage statistics (see docs). Additionally, per-thread CPU usage metrics are available to help detect high IO thread utilization (see PR). These metrics can be visualized in Grafana dashboards for real-time monitoring. Would you like more detail on which specific metrics to look for or how to set up Prometheus/Grafana for this purpose? Sources: - Add per-thread CPU usage metrics - see docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM pulsar_storage_ledger_write_latency_sum

The metric pulsar_storage_ledger_write_latency_sum represents the total sum of ledger write latencies (in milliseconds) for a topic, namespace, or subscription, as recorded by the broker. It accumulates the total time spent writing entries to BookKeeper ledgers (see reference). This metric is useful for calculating average ledger write latency when divided by pulsar_storage_ledger_write_latency_count. Would you like more detail on how to use this metric or how to interpret it? Sources: - Pulsar resource metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM Unable to read fd field from java.io.FileDescriptor

The error "Unable to read fd field from java.io.FileDescriptor" occurs when BookKeeper or Pulsar code tries to access the internal file descriptor using Java reflection and fails, often due to Java module restrictions or security settings (see PageCacheUtil.java). This is common on JDK 11+ or JDK 17+ unless the JVM is started with the appropriate --add-opens options to allow reflective access (see Pulsar PR). To resolve, add the following JVM options: --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/sun.nio=ALL-UNNAMED Would you like more detail on where to set these options or why this access is needed? Sources: - PageCacheUtil.java - remove illegal access warning and enable reads optimization on jdk17 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM [BookieJournal-3181] WARN org.apache.bookkeeper.util.NativeIO - Unable to read fd field from java.io.FileDescriptor","pulsar-bookkeeper", in bookie log .what is the problem

This warning means BookKeeper tried to access the internal file descriptor (fd) of a java.io.FileDescriptor object using reflection, but failed due to Java module access restrictions, especially on JDK 11+ or JDK 17+ (see PageCacheUtil.java). This prevents certain native IO optimizations, but BookKeeper will continue running, just without those optimizations. To fix, add JVM options: --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/sun.nio=ALL-UNNAMED (see Pulsar PR). Would you like more detail on how to apply these options? Sources: - PageCacheUtil.java - remove illegal access warning and enable reads optimization on jdk17 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.