Hi team, We have 3 brokers(broker01,02,03) and 3 bookkeeper(bookie01,02,03) VM, all were running apache pulsar 2.8. We are in the process of upgrading to 3 along with OS upgrade. We will need to decommission bookies before os upgrade/ vm reprovision. While doing the decom bookie, after a point it stuck at 1 and never finishes. On checking we found the ledger under replicated and showing quorum 3.

bookkeeper shell listunderreplicated | grep ListUnderReplicatedCommand | grep -v Ctime | awk -F- '{print $2}'
 1

LedgerMetadata{formatVersion=3, ensembleSize=3, writeQuorumSize=3, ackQuorumSize=3, state=CLOSED, length=1000, lastEntryId=9, digestType=CRC32C, password=base64:, ensembles={0=[172.16.17.232:3181, 172.16.17.229:3181, 172.16.17.231:3181]}, customMetadata={}}

Whereas all other ledger having quorum 2 and is replicated. What can be done here to complete the decommission or change quorum to 2.

Try doing a rolling upgrade to ensure that you always have 3 bookie instances, otherwise quorum will be broken and you will this error

that's difficult, we have only 3 and need to take one down for the upgrade

Check out Sections 2.1.2 and 2.1.3 of this blog.