@RunLLM Where is numIoThreads, numListenerThreads and connectionPerBroker used for in the pulsar client config and how are they related? What should improve when increasing numIoThreads and connectionPerBroker and how can I define a correct value?

numIoThreads

sets the number of threads for handling network connections to brokers, while

numListenerThreads

sets the number of threads for message listeners (used by consumers/readers with a listener model).

connectionsPerBroker

controls the maximum number of TCP connections the client opens to each broker. These are independent settings:

numIoThreads

and

numListenerThreads

affect concurrency and parallelism, while

connectionsPerBroker

affects connection pooling and throughput per broker. By default, all are set to 1 or the number of available processors, depending on the client and language (Pulsar client docs, Java ClientBuilder, ClientConfigurationData.java). Increasing

numIoThreads

can improve throughput if you have many producers/consumers or high network load, and increasing

connectionsPerBroker

can help when many producers/consumers share a broker, especially over high-latency networks. However, setting these too high can waste resources or cause contention. Start with defaults, monitor performance, and increase only if you observe bottlenecks (Flink Pulsar connector docs, Pulsar client config reference). Would you like more detail or code examples? Sources: - Java ClientBuilder - Pulsar client docs - ClientConfigurationData.java - Pulsar client config reference 👍 1 button 👎 0 button 💡 Hint: Mention @RunLLM in the thread for followups.

@RunLLM We see frequent connects and reconnects happening in our pulsar client, is this normal behaviour? And does it impact performance?

Frequent connects and reconnects are not normal in a healthy Pulsar deployment and can indicate network instability, broker issues, or misconfiguration. This behavior can negatively impact performance by increasing latency, causing message send failures, and potentially leading to backlogs or timeouts (Pulsar binary protocol docs, pulsar-client-go issue #1027). If you observe this, investigate network reliability, broker health, and client configuration (timeouts, keepalive, etc.). Persistent reconnects can cause message loss or delays, so resolving the root cause is important for stable performance. Would you like more detail on troubleshooting steps? Sources: - pulsar-client-go issue #1027 - Pulsar binary protocol docs 👍 1 button 👎 0 button 💡 Hint: Mention @RunLLM in the thread for followups.

@RunLLM We see frequent connects and reconnects happening in our pulsar client, is this normal behaviour? And does it impact performance? We use the pulsar-client-java. Does is close idle connections and than just reconnect when there is work or is the connection always there?

Frequent connects and reconnects are not normal and may indicate network, broker, or configuration issues; this can negatively impact performance by increasing latency and causing message send failures (Pulsar binary protocol docs). The Pulsar Java client can close idle connections if

connectionMaxIdleSeconds

is set (default 180s); when a new operation is needed, it will reconnect as required (Flink Pulsar connector docs, PR #15490). If you see frequent reconnects even when there is active work, investigate network stability and broker health. Idle connection cleanup is normal, but excessive reconnects during active use are not. Would you like more detail on configuration or troubleshooting? Sources: - Flink Pulsar connector docs - PR #15490 - Pulsar binary protocol docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM So connectionPerBroker and numIoThreads are related? How do they correlate and how to set them?

connectionsPerBroker

sets how many TCP connections the client opens to each broker, while

numIoThreads

sets the number of threads used to handle those connections. They are related in that more connections may require more I/O threads for optimal performance, but they are configured independently (Java ClientBuilder, Flink Pulsar connector docs). If you increase

connectionsPerBroker

(e.g., for higher throughput or many producers/consumers), consider increasing

numIoThreads

to avoid thread contention. Start with defaults, monitor performance, and tune upwards only if you see bottlenecks. Would you like more detail or tuning guidelines? Sources: - Java ClientBuilder - Flink Pulsar connector docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM So do you need to keep them equal?

It seems likely that you do not need to keep

connectionsPerBroker

and

numIoThreads

equal; they can be tuned independently based on workload and system resources (Java ClientBuilder, Flink Pulsar connector docs). The defaults are typically sufficient, and only in high-throughput or high-concurrency scenarios might you need to increase both, but not necessarily to the same value. Monitor your system: if you see I/O thread contention, increase `numIoThreads`; if you see connection bottlenecks, increase

connectionsPerBroker

. Would you like more detail on how to monitor or tune these settings? Sources: - Java ClientBuilder - Flink Pulsar connector docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.