Is there a way to spread the replicas per partitio...
# troubleshootingm
Matt
02/18/2021, 5:33 PMIs there a way to spread the replicas per partition to different AZs? I would like the replicas to be in a different host on different AZ for HA.
n
Nick Bowles
02/18/2021, 5:43 PMYou’re using the helm chart I assume? What cloud provider?
s
Subbu Subramaniam
02/18/2021, 5:44 PMMaybe you should also chime in on https://github.com/apache/incubator-pinot/issues/6532 ?
m
Matt
02/18/2021, 5:58 PMI am using AWS
Matt
02/18/2021, 6:00 PMLooks like the PR is what I need, seems not much progress though
s
Subbu Subramaniam
02/18/2021, 6:08 PMWhat is needed is a generic Pinot mechanism to support the criteria that each replica should be in a separate <insert your cloud-dependent term here>. This requires pinot to interact with the cloud APIs to fetch/parse/store some information in a cloud-independent fashion.
n
Nick Bowles
02/18/2021, 6:50 PMIf you’re using k8s, I think this can be solved on the k8s/cloud provider side by doing this:
• Set nodegroups for server to be multi AZ
• Use labels/podAntiAffinity to keep server pods from being colocated OR set it up as a daemonset so there will always be 1 server pod per node
• Set your replications depending on # of nodes (or if there is a way to dynamically set this)
Nick Bowles
02/18/2021, 6:51 PMIt’s hacky but might work to achieve HA
m
Matt
02/18/2021, 6:54 PMThe third point might not work as I am planning to have 9 instances. 3 in each AZ. There is a chance that all replicas end up in same AZ.
Matt
02/18/2021, 6:58 PM@Subbu Subramaniam I can label servers based on AZ. So all Pino have to ensure is to not create replica in same label server.
s
Subbu Subramaniam
02/18/2021, 7:08 PMCan you add these ideas to the issue? thanks. A solution we come up with should be cloud generic.
n
Nick Bowles
02/18/2021, 7:20 PMI’ll add some things in there in a bit and think through it some more. Something is blowing up at work right now and I’m the only person who can do SRE 😅