@RunLLM does geo-replication has any retries, if replication disabled by broker? As i see, when i get some load increase, replication stoped and doesn't continued. Only unset clusters and set again helped.

If geo-replication is disabled by the broker (for example, if the replication clusters are removed from the namespace or topic), Pulsar will stop replicating messages to the remote cluster. Replication will not automatically retry or resume unless you re-enable replication by setting the clusters again. There is no built-in retry mechanism for geo-replication if it is administratively disabled; replication resumes only when you explicitly reconfigure the clusters for replication according to Pulsar documentation and namespace replication API. If replication stops due to load or broker issues (not administrative disablement), Pulsar will attempt to resume replication when the issue is resolved. But if replication is disabled in the configuration, only manual intervention (resetting clusters) will restore it. Would you like more detail on how to monitor or troubleshoot geo-replication failures? Sources: - namespace replication API 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM I ran into the following issue. At some point, the load on one of the topics increased. The replication stopped spontaneously, and there were no attempts to reconnect to the remote cluster. Initially, I tried unloading the topic on both clusters, but that didn’t help. Then I restarted the proxies and brokers on both clusters. After restarting the brokers, replication for a few more topics failed. As a result, only 90 out of 93 topics were being replicated. After 10 minutes, replication for the remaining 3 topics still hadn’t recovered, so I disabled replication on the source cluster and then re-enabled it (using the set-clusters command). What could be the root cause, and how can I add retries for replication?

The root cause is likely that the replicator for those topics entered a failed state due to the load spike or a transient error, and Pulsar did not automatically recover or retry replication for those topics. Pulsar's geo-replication does not have a built-in retry or self-healing mechanism if the replicator fails due to broker or network issues; it only resumes if the topic is reloaded or replication is reconfigured (e.g., via set-clusters) as you did. This is a known operational gap—see issue 24819 and related discussions. Currently, there is no configuration to add automatic retries for failed replicators. Manual intervention (unloading topics, restarting brokers, or toggling replication clusters) is required to recover. Would you like more detail on monitoring or scripting recovery for failed geo-replication? Sources: - issue 24819 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM unloading topics, restarting brokers doesn't help

Unloading topics and restarting brokers may not recover stuck geo-replication because the replicator can remain in a failed state if the underlying issue (such as a network or metadata inconsistency) persists. As reported in issue 24819, only disabling and re-enabling replication with the set-clusters command reliably restarts replication for affected topics. There is currently no automatic retry or self-healing for failed replicators in Pulsar; manual intervention is required. Would you like more detail on possible workarounds or monitoring strategies? Sources: - issue 24819 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM i get some error before disabling replication Error: Too many requests to the same Bookie while reading

The error "Too many requests to the same Bookie while reading" indicates that the BookKeeper client is overloading a bookie with read requests, exceeding its configured limits. This can happen during high load or backlog catch-up, and may cause replication or topic operations to stall or fail, as seen in BookKeeper issue 2945. This error is not automatically retried by Pulsar's geo-replication; the replicator may get stuck until manual intervention. Would you like more detail on tuning BookKeeper or Pulsar to avoid this error? Sources: - BookKeeper issue 2945 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM and this one Bookie has been quarantined because of read/write errors.

The message "Bookie has been quarantined because of read/write errors" means Pulsar's BookKeeper client detected repeated failures with a bookie and temporarily excluded it from new ensembles, as described in BookieWatcherImpl.java. This is a safety mechanism to avoid routing traffic to unhealthy bookies. Quarantined bookies are not retried for new writes/reads until the quarantine period expires or all bookies are quarantined, at which point Pulsar may attempt to use them again. Would you like more detail on how to adjust quarantine behavior or recover from this state? Sources: - BookieWatcherImpl.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yes please. And please give me needed config for bookie with 50k per seconds incoming/outcoming messages and needed resources. Input/output speed 8mb/s

It seems likely that to adjust quarantine behavior, you can tune the following Pulsar broker configs:

bookkeeperClientHealthCheckErrorThresholdPerInterval

(number of errors before quarantine),

bookkeeperClientHealthCheckQuarantineTimeInSeconds

(quarantine duration), and

bookkeeperClientTimeoutInSeconds

(timeout for BookKeeper operations). These are set in

broker.conf

as described in ServiceConfiguration.java. Lowering the error threshold or quarantine time can make recovery faster, but may increase risk if a bookie is truly unhealthy. For a bookie handling 50,000 messages/sec and 8 MB/s throughput, the hardware recommendations suggest at least 4 vCPUs, 8 GB RAM, SSD journal (256 GB), and 2 TB HDD for ledgers. This matches the recommended config for high-throughput clusters. Ensure your network and disk I/O can sustain the required throughput. Would you like a sample configuration file or more detail on tuning these parameters? Sources: - ServiceConfiguration.java - Pulsar hardware recommendations 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yes, please give me more tuning parameters, with JVM options and bookie config, for caches and rocksDB params

My understanding is that for high-throughput bookies (50k msg/s, 8MB/s), you should tune both JVM and BookKeeper/rocksDB parameters. For JVM, set

-Xms

and

-Xmx

to 4-8g (or higher if RAM allows), and set

-XX:MaxDirectMemorySize

to at least 4g. Use G1GC:

-XX:+UseG1GC

. See StreamNative JVM tuning guide. For BookKeeper, in

bookkeeper.conf

, set: -

dbStorage_writeCacheMaxSizeMb

and

dbStorage_readAheadCacheMaxSizeMb

to 1024 (1GB) or higher if RAM allows. -

dbStorage_rocksDB_blockCacheSize

to 10% of direct memory (e.g., 409600000 for 4GB). -

dbStorage_rocksDB_writeBufferSizeMB

to 128 or 256. -

dbStorage_rocksDB_sstSizeInMB

to 128. -

dbStorage_rocksDB_numLevels

to 6. -

dbStorage_rocksDB_numFilesInLevel0

to 8. -

dbStorage_rocksDB_maxSizeInLevel1MB

to 512. These values are based on defaults and tuning advice in BookKeeper issue 3507, Pulsar PR 23178, and StreamNative JVM docs. Adjust upward if you have more RAM and monitor for GC or memory pressure. Would you like a sample config file or more details on monitoring? Sources: - BookKeeper issue 3507 - Pulsar PR 23178 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM for this setup how many ram i need?

It seems likely that for a bookie handling 50,000 messages/sec and 8 MB/s throughput, you should allocate at least 8 GB RAM per bookie node, as recommended for high-throughput clusters in the Pulsar hardware guide. This allows for sufficient JVM heap, direct memory, and RocksDB/block caches. If you increase cache sizes or RocksDB buffers beyond defaults, or expect higher spikes, 16 GB RAM per bookie is safer. Would you like more detail on how to size each memory component? Sources: - Pulsar hardware recommendations 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM if i set 4gb xmx/xms and 4g DirectMem how many ram i need? How many non heap mem i need?

My understanding is that if you set

-Xms4g -Xmx4g

(heap) and

-XX:MaxDirectMemorySize=4g

(direct memory), you need at least 8 GB RAM for the JVM process alone. However, you should add extra headroom (2–4 GB) for non-heap memory (e.g., code cache, thread stacks, native libraries, and RocksDB native memory), so a minimum of 10–12 GB RAM per bookie is recommended for stability, as also suggested in BookKeeper tuning discussions. Non-heap memory includes direct memory, thread stacks, code cache, and native allocations (like RocksDB). If you use 4 GB heap and 4 GB direct memory, plan for at least 2–4 GB extra for these non-heap usages. Would you like a breakdown of each memory component? Sources: - BookKeeper tuning discussions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.