<@U078Y59KC3B> Can you help me diagnose this error...
# ask-ai
s
@RunLLM Can you help me diagnose this error on Pulsar Recovery pods: "Failed on operating ledger metadata: -25"
r
The error message "Failed on operating ledger metadata: -25" indicates a
NoSuchLedgerExistsOnMetadataServerException
in Apache BookKeeper, which is part of the Pulsar ecosystem. This error typically occurs when the system attempts to access a ledger that does not exist in the metadata store. This can happen due to several reasons, such as: 1. Ledger Deletion: The ledger might have been deleted from the metadata store, but references to it still exist in the system. 2. Metadata Inconsistency: There might be inconsistencies between the metadata stored in ZooKeeper and the actual data in BookKeeper. 3. Network Issues: Temporary network issues might have caused a failure in accessing the metadata store. To resolve this issue, you can try the following steps: 1. Check Ledger Existence: Use BookKeeper shell commands to verify if the ledger exists. You can use
bin/bookkeeper shell listledgers
to list all ledgers and check if the problematic ledger is present. 2. Metadata Consistency Check: Use tools like PCK (Pulsar Consistency Checker) to detect and resolve inconsistencies between Pulsar metadata and data. PCK can help identify missing or orphan ledgers (StreamNative PCK Overview). 3. Review Logs: Check the logs of the BookKeeper and Pulsar components for any additional error messages or warnings that might provide more context about the issue. 4. Network and Configuration: Ensure that there are no network issues affecting the connection to the metadata store and that the configuration settings for BookKeeper and ZooKeeper are correct. If the issue persists, you may need to manually intervene to correct the metadata or consult the community for further assistance. Sources: - BookKeeper Issue #3974 - StreamNative PCK Overview 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
s
@RunLLM The error is persistent and is for a different ledger every time. When it first started it was preceded by an "Interrupted while replicating fragments" error. Is it possible a bookie pod data got corrupted when it restarted?
r
Yes, it's possible that a bookie pod's data got corrupted during a restart, which can lead to persistent errors like "Failed on operating ledger metadata: -25" and "Interrupted while replicating fragments." Here are some potential causes and steps to address the issue: 1. Data Corruption: If a bookie pod was abruptly restarted, it might have led to data corruption, especially if the bookie was in the middle of writing data. This can cause inconsistencies in the ledger metadata. 2. Ledger Metadata Issues: The error "Interrupted while replicating fragments" suggests that there might have been an issue during the replication process, possibly due to network interruptions or bookie failures. 3. AutoRecovery: If AutoRecovery is enabled, it might be trying to replicate data from a corrupted ledger, leading to repeated errors. According to BookKeeper Issue #3905, autorecovery can get stuck if there are not enough non-faulty bookies available. Steps to Address the Issue: 1. Check Bookie Logs: Review the logs of the affected bookie pods for any signs of data corruption or errors during the restart process. 2. Use BookKeeper Shell Commands: Use commands like
bin/bookkeeper shell localconsistencycheck
to check for inconsistencies in the local bookie data. This can help identify corrupted ledgers. 3. Manual Intervention: If specific ledgers are consistently causing issues, you might need to manually remove or repair them. This can involve using BookKeeper shell commands to delete problematic ledgers or using the
autoSkipNonRecoverableData
configuration to skip over non-recoverable data (Pulsar Issue #21751). 4. Data Recovery: If data recovery is necessary, consider adding new bookies to the cluster and using the
bin/bookkeeper shell recover
command to re-replicate data from healthy bookies to the new ones. 5. Prevent Future Corruption: Ensure that bookie pods are gracefully shut down and restarted to prevent data corruption. Consider using persistent volumes to maintain data integrity across restarts. If the issue persists, further investigation into the specific circumstances of the bookie restarts and the state of the cluster during those times may be necessary. Sources: - BookKeeper Issue #3905 - Pulsar Issue #21751 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.