Hi Team ,
Due to power outage 2 out of 3 bookie w...
# generals
sindhushree
05/08/2024, 5:23 AMHi Team ,
Due to power outage 2 out of 3 bookie where down and came back later on the running pulsar cluster .
But out of 500 topics one of the topic partition stats is giving below error and messages are not flowing to this topic .
Any thing i can check or any way i can recover this topic .
pulsar@itomdipulsar-bastion-0:/pulsar/bin> ./pulsar-admin topics stats persistent://public/default/profile1_9-partition-1
--- An unexpected error occurred in the server ---
Message: No such entry
Stacktrace:
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such entry
Reason:
--- An unexpected error occurred in the server ---
Message: No such entry
Stacktrace:
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such entry
sindhushree
05/16/2024, 4:53 AMOn further debugging in the broker logs we could see that read entry was failing .
Then we ran bookie ledger metadata and readledger below are the ouput .
metadata :
2024-05-16T044822,297 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - LedgerMetadata{formatVersion=3, ensembleSize=2, writeQuorumSize=2, ackQuorumSize=2, state=CLOSED, length=8336855, lastEntryId=474, digestType=CRC32C, password=base64:, ensembles={0=[itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181, itomdipulsar-bookkeeper-1.itomdipulsar-bookkeeper.chotest:3181], 475=[itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181, itomdipulsar-bookkeeper-0.itomdipulsar-bookkeeper.chotest:3181]}, customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=, pulsar/managed-ledger=base64:cHVibGljL2RlZmF1bHQvcGVyc2lzdGVudC9wcm9maWxlMV85LXBhcnRpdGlvbi0x, application=base64:cHVsc2Fy}}
sindhushree
05/16/2024, 4:53 AMread ledger :
ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L315020253 E474-E474, Sent to [itomdipulsar-bookkeeper-1.itomdipulsar-bookkeeper.chotest:3181, itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181], Heard from [] : bitset = {}, Error = 'No such entry'. First unread entry is (-1, rc = null)
sindhushree
05/16/2024, 4:54 AM@Penghui @sijieg Can you give us some hint or ways to recover this kind of ledger ?