This message was deleted.

The files using most of the space are in /home/pulsar/data/bookkeeper/ledgers/current

You can format the local filesystem data on a bookie using the

bookieformat

command, e.g.

$ bin/bookkeeper shell bookieformat

If your

metadataServiceUri

is pointing to the correct ZK cluster, then it should re-fetch the cookieID and join the cluster.

Aha. I already did a decom of the dead bookie on one of the other nodes: bin/bookkeeper shell decommissionbookie -bookieid xxx That is going to take quite some more time (10 hours+) to complete. Should I kill that and do as you suggest or let it run?

The decom process is offloading the data to maintain the replica count requirement, hence the 10+ hour duration. I would let it continue to run, as it is the preferred method for removing a node.

And then I can just start up the old bookie with empty journal and ledgers and it should come back? Or should I then do the bookieformat? Tx for the help.

In theory yes. And the auto recovery process should handle the under replicated segments for you

The machine running the decom crashed so I thought I would try the bookieformat. It doesn’t seem to do much (don’t see any files, it just makes the directories): $ bin/bookkeeper shell bookieformat 2023-07-31T192127,655+0000 [main] INFO org.apache.bookkeeper.bookie.Bookie - Bookie format completed successfully $ find data/bookkeeper/ data/bookkeeper/ data/bookkeeper/journal data/bookkeeper/ledgers

Are there any messages in the bookie log? Specifically ones indicating that it is connecting to the ZK cluster and registering itself as a node in the cluster?

I tried to start it up and it complained about the missing cookie again.

Does the bookie show up in the list of bookies if you run

bin/bookkeeper shell listbookies

?

Yes it does show up for listbookies, both on the dead node only running zk and on the others. I have the decom running again.

It should fetch the cookie after you start the bookie.

Aha. The decom finished and deleted the cookie. The old bookie started successfully and is replication ledgers. Thanks for all the help.

Glad to help

hello @David Tinker i too am facing this

There are directories without a cookie, and this is neither a new environment,

in my openshift cluster where i am trying to install pulsar i dont have any data for now that i want in persistent volume. so should i delete the old PVs or run the decom u mentioned?(could u also tell what this demon is) thanks in advance😄

Hmm I am not an expert on this .. I just decom’ed:

bin/bookkeeper shell decommissionbookie -bookieid 10.0.0.61:3181

On one of the working bookies. The bookieid there is the dead one.

how did u know the bookieid is it this command

bin/bookkeeper shell bookieformat --force

? and where should i run it if so. inside any pod or ..?

We are running on bare metal so the bookie ID is in the config files.

okayy, i am not familier with that . 1 last doubt, could u tell me what this issue means? like old BK had created some PV, and bcoz of that new BK cant be installed?

In our case the machine ran out of space so we wiped the data expecting it to replicate ledgers. But the bookie expects to find the cookie and won’t start. Maybe your bookie has started with a new PV for some reason?

yes, i had deleted the old BK because it was failing for some reason