Hi, the recommended deployment order for pinot is controller -> broker -> server -> minion, and I see on the docs that it's a recommendation rather than hard requirement. If I'm running pinot across 3 kubernetes clusters (each cluster with their own controllers/brokers/servers/minions), is there a requirement that the controller needs to be deployed across all 3 clusters, before any other component? meaning controller in first cluster, then second, then third clusters, and then broker/server/minion in other clusters? or can i do controller -> broker -> server -> minion in first cluster, then second, and then third?

This recommendation is to ensure zero-downtime upgrades. The project maintains new controller can talk to old components, similarly new broker can talk to old server, but not the other way round in case there is a communication protocol change. This applies to the scope of one cluster. You can deploy air gapped clusters independently

gotcha, deployment order makes sense. in a cluster that's undergoing a deploy, is there downtime if some controllers are upgraded, while others aren't? or does it not matter because only one controller is actually active and the others are passive? also, just to confirm, when you say air gapped clusters, are you referring to 3 pinot clusters that are each in their own kubernetes clusters and are independent? i'm more asking about deployment order if we have 1 pinot cluster deployed with components among 3 kubernetes clusters (such as 3 controllers, 3 brokers, etc for the pinot cluster, 1 in each k8s cluster, so that the entire pinot cluster is deployed across multiple AZs). can we follow the same deployment order, but sequentially across k8s clusters? is there any need to deploy all controllers first?

Yes, this needs to follow the deployment order, as it is a single pinot cluster. Also, this why do you need 3 k8s clusters for this (moreover, how does it even work)?

pinot cluster deployed with components among 3 kubernetes clusters (such as 3 controllers, 3 brokers, etc for the pinot cluster, 1 in each k8s cluster, so that the entire pinot cluster is deployed across multiple AZs).

we have 3 k8s clusters to be across a few AZs, and they are all registered with the same helix / zk configs

You can use single cluster and map an AZ to server pool to achieve the same