@RunLLM our Bookkeepers went down because they ran out of disk space. How can I recover the cluster now?

To recover BookKeeper bookies that went down due to running out of disk space, you should first free up space (e.g., by increasing disk size, cleaning up old logs, or reducing retention). Once space is available, restart the bookie pods. If the bookies still fail to start, you may need to adjust configuration parameters like

forceReadOnlyBookie

, reduce log file sizes, or disable entry log preallocation. If you add new disks or directories, you must update the BookKeeper cookie using the documented procedure to reflect the new storage layout. For persistent issues, check for disk or data corruption and consider using manual recovery or updating cookies as described in the documentation and community discussions. For step-by-step recovery, see the procedure outlined in this issue comment, which includes resizing volumes, updating cookies, and restarting bookies. If you need more detail, let me know. Sources: - How to recover Bookies that completely run out of storage space? - Bookie 99% ledger disk usage 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how can I clear corrupted ledgers from Bookkeeper nodes?

To clear corrupted ledgers from BookKeeper nodes, you can use the

bookkeeper shell

tool to identify and delete problematic ledgers. First, use commands like

readlog

or

localconsistencycheck

to detect corrupted ledgers. Then, delete the identified ledgers with the

deleteledger

command. Always ensure you understand the impact, as deleting ledgers may result in data loss for topics using those ledgers. For more details on identifying and deleting ledgers, see this issue discussion. Would you like a step-by-step example? Sources: - Bookkeeper data logs sometimes get ERROR - Failed to compact entry log 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM I am seeing

2025-08-01T07:38:35,263+0000 [BookieReadThreadPool-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.proto.ReadEntryProcessorV3 - IOException while reading entry: 0 from ledger 11
java.io.IOException: Bad entry read from log file id: 0

. Should I delete the ledger if I see this issue?

If you see

IOException: Bad entry read from log file id: 0

when reading a ledger entry, it indicates data corruption in the entry log. Deleting the ledger will remove the corrupted data, but this will also result in permanent data loss for any topic using that ledger. Deletion is a last resort if the ledger is unrecoverable and you accept the data loss. Before deleting, you should try to confirm the corruption using

bookkeeper shell readlog

or

localconsistencycheck

as described in this issue. If confirmed and the ledger is blocking cluster operations, deletion is a valid recovery step. Would you like detailed deletion steps? Sources: - Bookkeeper data logs sometimes get ERROR - Failed to compact entry log 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM Yes, please share detailed deletion steps

To delete a corrupted ledger from BookKeeper, follow these steps: 1. Identify the Ledger ID: Confirm the problematic ledger ID (e.g., 11) using logs or

bookkeeper shell readlog

to verify corruption. 2. Delete the Ledger: Use the BookKeeper shell command:

bin/bookkeeper shell deleteledger --ledgerid <ledgerId> --force

Replace

<ledgerId>

with your actual ledger ID (e.g., 11). The

--force

flag skips confirmation prompts. 3. Verify Deletion: Optionally, check that the ledger is removed using:

bin/bookkeeper shell listledgers | grep <ledgerId>

If the ledger does not appear, it has been deleted. For more on the delete command, see DeleteLedgerCommand.java. Would you like guidance on handling metadata or Pulsar recovery after deletion? Sources: - DeleteLedgerCommand.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.