Hi team, one question about highly available pinot setup. Assuming we deploy one pinot cluster to 3 availability zone, and we have table that has 3 replicas per segment, is there anyway to enforce pinot distribute the table segment replicas to 3 servers in different availability zone? Want to see if there are some example to achieve this with server tagging or something.

Unfortunately Pinot is not currently AZ aware. This may be a feature provided later in 2023.

In the interim if you intend to do so, you could achieve this manually: • Create server-pools • Map server in one server pool to one AZ Once this pool-AZ mapping is established, rest of the system will honor the pool mapping, and hence be implicitly AZ aware.

so if I have 3 server pool for 3 AZ, I can enforce the table to assign the 3 replica to 1 pool each?

The replicas will be auto assigned as long as server pooling is set up wherever needed

can you point me to some documentation around this segment <> server pool assignment

trying to capture the full story, is it working in this way? 1. create a set a servers in one AZ( a k8s staefulset for example) 2. assign pool_tag to servers created in step 1; 3. repeat 1-2 if we have replica>1; 4. assign pool tag in table_definition Another question is, what about multi-AZ for controller/broker?

found this part of the doc regarding server pool x replica group, seems useful: https://docs.pinot.apache.org/operators/operating-pinot/instance-assignment#pool-based-instance-assignment

i think the fault domain aware assignment at the bottom is supposed to do what you want assuming you correctly map each AZ to a single pool number. the design doc had examples explaining why this worked better than just replica group + AZ<>pool number mapping

Yes that is correct @Johan Adami ^^, what might be missing there is cloud vendor specific impl on annotating AZ info on instances.