We encountered this issue in 2 of our pulsar clusters where the bookie suddenly crashed and is failing on restarts. Below are some of the errors present in log. Any idea what is wrong here? When we checked bookies on zookeeper-shell, there was empty list.

The first two screenshots indicate that there is missing data in the Zookeeper node. Did you make any changes to ZK that might have erased some data?

No changes done to Zookeeper from our side. We spawned this pulsar cluster for testing purpose. We are chaos testing different components, as part of which we stopped the bookie process on this instance. Things ran well with one less bookie in the cluster. Now we are trying to start this stopped bookie again(after a couple of days of cluster being idle), in that case it is giving us this exception where the zkeeper data is missing. Not sure why it is happening without anyone manually doing it. 😶

Any config changes in the bookie? Is it pointing to the same ZK cluster?

No changes at all to the cluster apart from stopping a bookkeeper. And yes it is pointing to same zk cluster.

If you query the ZK node the bookie connects to, can you see an entry under

/ledgers/LAYOUT

?

Hi @David K, I checked the things pointed by you. Attached is the screenshot of data present under

/ledgers/LAYOUT

on ZK node

ip-10-10-17-248.ec2.internal

I added couple of new brokers to the cluster with same zookeeper ips, they got added successfully without any issues.

It appears that this particular bookie’s data was corrupted. I recommend removing it from the cluster and replacing it with a new one.

Yeah, done that for time being. Will update here if we face it again. Thank you for the guidance @David K 🙌