Once a bookie has been initialized, you cannot move it from an AZ to another. The pod gets scheduled where the PVC is

It's pretty painful to fix an existing deployment to handle AZs properly.

Moreover are there any delimiters which we need to set as part of the ZoneAwareEnsemblePolicy while setting up racks? We used the delimiter as

/region/az

(reference from the source code test cases) From the Source code, https://github.com/apache/bookkeeper/blob/df4492012cc03682534cbc8dd68dd81163b0c947[…]va/org/apache/bookkeeper/replication/TestReplicationWorker.java We are getting the below exception

Failed to perform http post request: javax.ws.rs.ClientErrorException: HTTP 412 {"reason":"Bookie 'rack' parameter is invalid, When `RackawareEnsemblePlacementPolicy` is enabled, the rack name is not allowed to contain slash (`/`) except for the beginning and end of the rack name string. When `RegionawareEnsemblePlacementPolicy` is enabled, the rack name can only contain one slash (`/`) except for the Bookie 'rack' parameter is invalid, When `RackawareEnsemblePlacementPolicy` is enabled, the rack name is not allowed to contain slash (`/`) except for the beginning and end of the rack name string. When `RegionawareEnsemblePlacementPolicy` is enabled, the rack name can only contain one slash (`/`) except for the beginning and e Reason: Bookie 'rack' parameter is invalid, When `RackawareEnsemblePlacementPolicy` is enabled, the rack name is not allowed to contain slash (`/`) except for the beginning and end of the rack name string. When `RegionawareEnsemblePlacementPolicy` is enabled, the rack name can only contain one slash (`/`) except for the beginning and end of the rack name string.

From the exception, they were mentioning RackawareEnsemblePlacementPolicy and RegionawareEnsemblePlacementPolicy,But they were nowhere mentioned about the ZoneawareEnsemblePlacementPolicy Latter, we tried with

az

as rack and it wont throw any exception, But it is not honouring these parameters

bookkeeperClientMinNumZonesPerWriteQuorum: "2"
  bookkeeperClientEnforceStrictZoneawarePlacement: "true"

And landed all the bookies' ledgers on the same availability zone Below is the rack information for ZoneawarePacementPolicy:

{
  "westus2": {
    "bookie-0": {
      "rack": "westus2-2"
    },
    "bookie-1": {
      "rack": "westus2-1"
    },
    "bookie-2": {
      "rack": "westus2-1"
    },
    "bookie-3": {
      "rack": "westus2-3"
    },
    "bookie-4": {
      "rack": "westus2-2"
    },
    "bookie-5": {
      "rack": "westus2-3"
    },
    "bookie-6": {
      "rack": "westus2-2"
    }
  }
}

where are these coming from:

bookkeeperClientMinNumZonesPerWriteQuorum: "2"
  bookkeeperClientEnforceStrictZoneawarePlacement: "true"

I don't see these referenced in apache/pulsar or apache/pulsar-helm-chart

which extends

TopologyAwareEnsemblePlacementPolicy

PULSAR_PREFIX_bookkeeper_bookkeeperClientEnforceStrictZoneawarePlacement

in helm chart for broker configmap

btw. IIRC, I believe that you should be able to use RegionAwareEnsemblePlacementPolicy to configure AZs too.

@swamy mavuri I'd recommend using "Region-aware placement policy" although it's really about AZs. In that case, you could simply follow the documentation in https://pulsar.apache.org/docs/next/administration-isolation-bookie/#region-aware-placement-policy to make the configuration.

Just want to summarise the learnings we had on the rack aware policies so that others can refer in future: 1.Pulsar doesn't support ZoneAwarePlacementPolicy, but bookkeeper does(https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ZoneawareEnsemblePlacementPolicy.java), (We need to query the pods and corresponding AZ from K8s to form the racks) and also it is advisable to form the scalesets while forming bookie nodepool, so that cloud provider will provide the placement for fault and upgrade domains 2.Pulsar supports Region Aware, but found few anomalies when we are evaluating.(https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/o[…]pache/bookkeeper/client/RegionAwareEnsemblePlacementPolicy.java) configured bookies on different region and it is working as usual. But with region aware, (when disaster happens)we are seeing an exception only after the ledger rollover happens, not immediately when there are not enough bookies with in the rack. lets say in region 1: bk1,bk2,bk3 region2: bk4 region3: bk5 (Here each region belongs to each rack) If i bring down the bookies in region 2 and 3. Immediately it is not throwing any exception and the ensemble formed on the region 1 bookies i.e., bk1,bk2,bk3(Though we set rack enforcement on the broker). https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/o[…]pache/bookkeeper/client/RegionAwareEnsemblePlacementPolicy.java (ReplaceFromRack method assigns bookie from the same rack i believe)