Sean Mannion
04/03/2025, 12:09 PMRunLLM
04/03/2025, 12:09 PMNoSuchLedgerExistsOnMetadataServerException in Apache BookKeeper, which is part of the Pulsar ecosystem. This error typically occurs when the system attempts to access a ledger that does not exist in the metadata store. This can happen due to several reasons, such as:
1. Ledger Deletion: The ledger might have been deleted from the metadata store, but references to it still exist in the system.
2. Metadata Inconsistency: There might be inconsistencies between the metadata stored in ZooKeeper and the actual data in BookKeeper.
3. Network Issues: Temporary network issues might have caused a failure in accessing the metadata store.
To resolve this issue, you can try the following steps:
1. Check Ledger Existence: Use BookKeeper shell commands to verify if the ledger exists. You can use bin/bookkeeper shell listledgers to list all ledgers and check if the problematic ledger is present.
2. Metadata Consistency Check: Use tools like PCK (Pulsar Consistency Checker) to detect and resolve inconsistencies between Pulsar metadata and data. PCK can help identify missing or orphan ledgers (StreamNative PCK Overview).
3. Review Logs: Check the logs of the BookKeeper and Pulsar components for any additional error messages or warnings that might provide more context about the issue.
4. Network and Configuration: Ensure that there are no network issues affecting the connection to the metadata store and that the configuration settings for BookKeeper and ZooKeeper are correct.
If the issue persists, you may need to manually intervene to correct the metadata or consult the community for further assistance.
Sources:
- BookKeeper Issue #3974
- StreamNative PCK Overview 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.Sean Mannion
04/03/2025, 12:25 PMRunLLM
04/03/2025, 12:25 PMbin/bookkeeper shell localconsistencycheck to check for inconsistencies in the local bookie data. This can help identify corrupted ledgers.
3. Manual Intervention: If specific ledgers are consistently causing issues, you might need to manually remove or repair them. This can involve using BookKeeper shell commands to delete problematic ledgers or using the autoSkipNonRecoverableData configuration to skip over non-recoverable data (Pulsar Issue #21751).
4. Data Recovery: If data recovery is necessary, consider adding new bookies to the cluster and using the bin/bookkeeper shell recover command to re-replicate data from healthy bookies to the new ones.
5. Prevent Future Corruption: Ensure that bookie pods are gracefully shut down and restarted to prevent data corruption. Consider using persistent volumes to maintain data integrity across restarts.
If the issue persists, further investigation into the specific circumstances of the bookie restarts and the state of the cluster during those times may be necessary.
Sources:
- BookKeeper Issue #3905
- Pulsar Issue #21751 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.