@RunLLM [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 2 bookies : excludeBookies [], allBookies []. 2026-02-25T045409,503 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger with ensembleSize=2, writeQuorumSize=2 and ackQuorumSize=2

This error means there are not enough available BookKeeper bookies to satisfy the requested ensemble, write quorum, and ack quorum sizes (all set to 2). The log shows that the set of available bookies is empty, so BookKeeper cannot create a new ledger or write data. At least two healthy bookies are required for this configuration, but none are available or writable at the moment according to BookKeeper administration docs. To resolve this, ensure at least two bookies are running and healthy in your cluster. You can check bookie status using the

bookkeeper shell listbookies -rw

command as described in the BookKeeper admin guide. Would you like more detail on troubleshooting or recovering bookies? Sources: - BookKeeper administration | Apache BookKeeper 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM have 3 bookies in RW

If you have 3 bookies in ReadWrite (RW) mode but still see "Not enough bookies to create ledger with ensembleSize=2, writeQuorumSize=2 and ackQuorumSize=2", it suggests that BookKeeper is unable to select bookies due to additional constraints, such as rack/region placement policies, bookie isolation groups, or health check quarantines. Even with 3 RW bookies, these policies can prevent ensemble formation if, for example, not enough racks are available or bookies are quarantined, as described in Pulsar rack-aware placement docs. Check for enforced rack/region policies, bookie isolation settings, or recent bookie failures/quarantines. Would you like more detail on how to diagnose these constraints? Sources: - Enforced minimum rack size of bookies 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM share the rack/region policies

BookKeeper supports three main data placement policies: rack-aware, region-aware, and zone-aware. - Rack-aware: Replicas are placed on different racks to ensure rack-level disaster tolerance. - Region-aware: Replicas are placed across different regions, and within each region, across different racks, for region-level disaster tolerance. - Zone-aware: Used in public cloud, places replicas across availability zones. You can configure these policies using the

ensemblePlacementPolicy

and related settings in your configuration files. For example, to enable region-aware placement, set

ensemblePlacementPolicy=org.apache.bookkeeper.client.RegionAwareEnsemblePlacementPolicy

and configure

reppDnsResolverClass

,

minNumRacksPerWriteQuorum

, and

enforceMinNumRacksPerWriteQuorum

as needed. More details and configuration examples are available in the Pulsar documentation and BookKeeper config reference. Would you like more detail or specific configuration examples? Sources: - Isolate bookies | Apache Pulsar - BookKeeper config reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM ls /bookkeeper/availablebookies nt showing

The path

/bookkeeper/availablebookies

is not a standard Zookeeper node used by BookKeeper to track available bookies. Instead, BookKeeper uses

/ledgers/available

in Zookeeper to store information about available bookies, as described in the BookKeeper admin documentation. To list available bookies, check the

/ledgers/available

znode in Zookeeper, not

/bookkeeper/availablebookies

. Would you like instructions on how to view this node or interpret its contents? Sources: - https://github.com/apache/pulsar/issues/23599 - https://github.com/apache/pulsar/issues/23807 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM am not seeing /ledgers/available is misisng on zk

If

/ledgers/available

is missing in ZooKeeper, BookKeeper cannot register or discover available bookies, which will prevent ledger creation and cause "Not enough bookies" errors. This znode is critical for BookKeeper operation and should be created during cluster initialization, as described in the BookKeeper admin docs. To fix this, re-initialize the BookKeeper metadata in ZooKeeper using the

bookkeeper shell metaformat

or

bookkeeper shell initnewcluster

command. Make sure all bookies are stopped before running these commands to avoid data loss. Would you like step-by-step instructions for this process? Sources: - https://github.com/apache/bookkeeper/issues/4031 - BookKeeper admin docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM on the working cluster this /ledgers/available/ went missing

If

/ledgers/available/

went missing on a working cluster, BookKeeper and Pulsar will not be able to discover or register bookies, leading to immediate write/read failures and "Not enough bookies" errors. This znode is essential for cluster health and should never be deleted during normal operation, as confirmed in BookKeeper admin docs. To recover, you must stop all bookies, re-initialize the BookKeeper metadata in ZooKeeper (using

bookkeeper shell metaformat

or

initnewcluster

), and then restart the bookies. Be aware: this can cause data loss if not done carefully. Would you like detailed recovery steps or guidance on minimizing data loss? Sources: - https://github.com/apache/pulsar/pull/23497 - https://github.com/apache/bookkeeper/issues/4287 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how to find out why that znode went missing

To find out why the

/ledgers/available

znode went missing, review recent user operations, automated scripts, or cluster management actions—such as running

bookkeeper shell metaformat

,

initnewcluster

, or manual deletions via ZK CLI—as these can delete critical znodes if misused. Also, check for Pulsar or BookKeeper bugs, misconfigurations, or network failures that could have triggered unintended metadata changes, as discussed in StreamNative PCK docs and BookKeeper issues. Audit logs, Zookeeper logs, and BookKeeper/Pulsar logs may provide clues about who or what deleted the znode. If audit logging is not enabled, it may be difficult to determine the exact cause after the fact. Would you like more detail on log analysis or prevention strategies? Sources: - BookKeeper Issue 4287 - StreamNative PCK docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM will the bookkeeper try to register with zk infinetly?

No, BookKeeper does not retry registration with ZooKeeper infinitely. If the ZooKeeper session expires or registration fails, BookKeeper will typically shut down rather than retry indefinitely, as recovering from a session expiration requires re-creating the ZooKeeper client and re-initializing all subcomponents. This is by design to allow external supervisors (like Kubernetes or supervisord) to restart the service, as explained in BookKeeper Issue 3250. Would you like more detail on the shutdown and recovery process? Sources: - BookKeeper Issue 3250 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM bookie process does not exit when zk connection fails

If the Bookie process does not exit when ZooKeeper connection fails, it is likely due to the shutdown logic not triggering a JVM exit after registration failure. This can happen if the shutdown method does not close the main thread's future or raise an exit signal, as discussed in BookKeeper Issue 4016. This is a known bug, and the expected behavior is for the Bookie to exit on ZooKeeper registration failure. Would you like more detail on possible workarounds or fixes? Sources: - BookKeeper Issue 4016 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.