@RunLLM pls help with error

--- An unexpected error occurred in the server --- Message: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6 Stacktrace: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6 at org.apache.pulsar.broker.service.BrokerService$2.openLedgerFailed(BrokerService.java:1899) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$10(ManagedLedgerFactoryImpl.java:466) at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(ManagedLedgerFactoryImpl.java:457) at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$initializeBookKeeper$2(ManagedLedgerImpl.java:583) at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6

The error "Not enough non-faulty bookies available error code: -6" means Pulsar cannot find enough healthy BookKeeper bookies to form a quorum for ledger operations. This usually happens if one or more bookies are down, in read-only mode, or not reachable, and your ensemble/write/ack quorum settings require more bookies than are available. Check the status of all bookies and ensure at least as many are healthy as your ensemble and quorum settings require. Restart or fix any failed bookies as needed. For more details on this error and troubleshooting steps, see this issue and the BookKeeper error code reference. Would you like a step-by-step guide to diagnose and resolve this in your cluster? Sources: - https://github.com/apache/pulsar/issues/23807 - https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/api/BKException.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM give me the commands to check and fix bookies

To check the status of your BookKeeper bookies, use:

bin/bookkeeper shell listbookies -rw -h   # Lists read-write (healthy) bookies
bin/bookkeeper shell listbookies -ro -h   # Lists read-only bookies

To test a bookie's health, run:

bin/bookkeeper shell bookiesanity

If bookies are in read-only mode or down, restart the affected bookie processes. If disks are full, free up space or add storage. For more, see BookKeeper CLI reference. Would you like detailed recovery or troubleshooting steps? Sources: - https://bookkeeper.apache.org/docs/reference/cli 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@Mustaque That means the writable bookies are not enough for your write replicas. How many replicas you are using and how many writable bookies on your cluster? You can list the bookies by using

bin/bookkeeper shell listbookies -rw

on bookie pod.