@RunLLM Bookkeeper admin api call returns success but the bookkeeper shell command fails with no such entry

A BookKeeper Admin API call (such as reading or listing ledgers) may return success if the metadata exists, but the bookkeeper shell command (like

readledger

) can fail with "no such entry" if the actual data is missing from the bookie storage. This typically happens when the ledger metadata is present in ZooKeeper, but the corresponding data entries are missing or corrupted in BookKeeper, often due to bookie failures, disk errors, or improper deletion operations. Tools like PCK can help detect such inconsistencies between metadata and data storage, identifying missing or orphaned ledgers according to topic metadata in ZooKeeper and BookKeeper storage systems. For more, see StreamNative PCK Overview. If you need to check for missing ledgers, use PCK or compare the results of

bookkeeper shell ledgermetadata

and

bookkeeper shell readledger

. If the metadata exists but reading entries fails, the data is likely missing or the ledger is corrupted. This is a known issue in BookKeeper and Pulsar environments, especially after bookie failures or network issues. See also Bookkeeper issue 4287 and Bookkeeper issue 3466. Would you like more detail on how to diagnose or resolve this inconsistency? Sources: - StreamNative PCK Overview - Bookkeeper issue 4287 - Bookkeeper issue 3466 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM bookkeeper read ledger admin call and bookkeeper shell read ledger is giving no such entry

If both the BookKeeper admin API (

readledger

) and the BookKeeper shell command (

bookkeeper shell readledger

) return "no such entry," it means the ledger metadata exists in ZooKeeper, but the actual data entries are missing from all bookies in the ensemble. This can occur due to bookie failures, disk corruption, or manual deletion of ledger data, leaving orphaned metadata behind. See Bookkeeper issue 4287 and Bookkeeper issue 3466. To confirm, check the ledger metadata with

bookkeeper shell ledgermetadata

and try reading the entries from each bookie in the ensemble. If all attempts fail, the data is lost or never written. Use tools like PCK to detect such inconsistencies between metadata and storage. Would you like step-by-step instructions for further diagnosis? Sources: - Bookkeeper issue 4287 - Bookkeeper issue 3466 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

yes

To diagnose "no such entry" errors for a ledger in BookKeeper, follow these steps: 1. Check Ledger Metadata Run:

bookkeeper shell ledgermetadata -l <ledger_id>

This confirms the ledger exists in ZooKeeper and shows the ensemble (list of bookies storing the ledger). 2. Attempt to Read Entries from Each Bookie For each bookie in the ensemble, try:

bookkeeper shell readledger -b <bookie_id> -l <ledger_id> -fe 0 -le <last_entry_id>

Replace

<bookie_id>

with each bookie from the ensemble, and

<last_entry_id>

with the value from the metadata. If all bookies return "no such entry," the data is missing or corrupted on all nodes. 3. Check for Orphaned or Missing Ledgers Use PCK or StreamNative tools to detect inconsistencies:

sn-pulsar-tools pck bookie detect-missing-ledger -z <zookeeper> -s <service-url> -t <topic>

This will list missing ledgers for a topic. See StreamNative PCK Tutorial. 4. Check for Orphaned Ledgers To find ledgers present in BookKeeper but not referenced by any topic:

sn-pulsar-tools pck bookie find-orphan-ledger -z <zookeeper> -s <service-url>

5. Resolution - If the ledger is unrecoverable, you may need to delete the metadata using PCK or manually clean up ZooKeeper. - If only some bookies are missing data, try BookKeeper's autorecovery or re-replication features. Would you like more detail on any specific step? Sources: - StreamNative PCK Tutorial - Bookkeeper issue 4287 - Bookkeeper issue 3466 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM bookkeeper admin api command to read ledger

The BookKeeper admin API command to read a ledger is

bookkeeper shell readledger -ledgerid <ledger_id>

. This command attempts to read all entries from the specified ledger and will return an error if the entries are missing or the ledger does not exist on the bookies. For more details, see BookKeeper 4.5.1 API docs. Would you like an example or more details on usage? Sources: - BookKeeper 4.5.1 API docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM -bookkeeper:8000/api/v1/ledger command to read specific entry

To read a specific entry from a ledger using the BookKeeper Admin REST API, use the following endpoint:

GET /api/v1/ledger/read/?ledger_id=<ledger_id>&start_entry_id=<start_entry_id>&end_entry_id=<end_entry_id>

For example, to read entry 5 from ledger 1234:

curl "http://bookkeeper:8000/api/v1/ledger/read/?ledger_id=1234&start_entry_id=5&end_entry_id=5"

This returns the content of entry 5 from ledger 1234. See BookKeeper Admin REST API for details. Would you like more information on the response format or error handling? Sources: - BookKeeper Admin REST API 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.