Takahiro Hozumi
08/23/2023, 4:17 AM$ docker exec -it pulsar_bookkeeper bin/bookkeeper shell listunderreplicated -printmissingreplica
...
2023-08-23T02:32:29,030+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.autorecovery.ListUnderReplicatedCommand - 4073123
2023-08-23T02:32:29,030+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.autorecovery.ListUnderReplicatedCommand - Ctime : 1692681280549
2023-08-23T02:32:29,030+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.autorecovery.ListUnderReplicatedCommand - MissingReplica : mybookie6:3282
...
$ docker exec -it pulsar_bookkeeper bin/bookkeeper shell ledgermetadata -ledgerid 4073123
...
2023-08-23T03:21:08,794+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - ledgerID: 4073123
2023-08-23T03:21:08,804+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - LedgerMetadata{formatVersion=3, ensembleSize=6, writeQuorumSize=2, ackQuorumSize=2, state=CLOSED, length=97726627, lastEntryId=49999, digestType=CRC32, password=base64:, ensembles={0=[mybookie1:3282, mybookie2:3282, mybookie3:3282, mybookie4:3283, mybookie5:3282, mybookie6:3282]}, customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=, pulsar/managed-ledger=base64:xxxxx, application=base64:cHVsc2Fy}}
...
bookkeeper log in autorecovery
$ sudo tail -f /var/lib/docker/containers/1551a5ad53c31504655d042563dbb151afee7d1a23b88b1d36a015d4ac4d3c31/1551a5ad53c31504655d042563dbb151afee7d1a23b88b1d36a015d4ac4d3c31-json.log
...
{"log":"2023-08-22T17:07:07,143+0000 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 4073123 for 1 number of times, so deferring the ledger lock release by 18750 msecs\n","stream":"stdout","time":"2023-08-22T17:07:07.144029311Z"}
...
{"log":"2023-08-23T04:02:48,920+0000 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [\u003cBookie:mybookie5:3282\u003e, \u003cBookie:mybookie4:3282\u003e, \u003cBookie:mybookie6:3282\u003e, \u003cBookie:mybookie1:3282\u003e, \u003cBookie:mybookie2:3282\u003e, \u003cBookie:mybookie3:3283\u003e], fallback to choose bookie randomly from the cluster.\n","stream":"stdout","time":"2023-08-23T04:02:48.920207638Z"}
{"log":"2023-08-23T04:02:48,920+0000 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [\u003cBookie:mybookie5:3282\u003e, \u003cBookie:mybookie4:3282\u003e, \u003cBookie:mybookie6:3282\u003e, \u003cBookie:mybookie1:3282\u003e, \u003cBookie:mybookie2:3282\u003e, \u003cBookie:mybookie3:3283\u003e], allBookies [\u003cBookie:mybookie4:3282\u003e, \u003cBookie:mybookie3:3283\u003e, \u003cBookie:mybookie1:3282\u003e, \u003cBookie:mybookie2:3282\u003e, \u003cBookie:mybookie5:3282\u003e].\n","stream":"stdout","time":"2023-08-23T04:02:48.920247312Z"}
{"log":"2023-08-23T04:02:48,920+0000 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment\n","stream":"stdout","time":"2023-08-23T04:02:48.920411679Z"}
{"log":"org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
{"log":"2023-08-22T17:07:07,143+0000 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 4073123 for 1 number of times, so deferring the ledger lock release by 18750 msecs\n","stream":"stdout","time":"2023-08-22T17:07:07.144029311Z"}