This message was deleted.
# troubleshootings
Slackbot
11/17/2022, 5:02 PMThis message was deleted.
c
Cory Johannsen
11/17/2022, 5:03 PMWhat specifically do you mean by "multi region"?
d
Didip Kerabat
11/17/2022, 5:04 PMSetup 2 Druid clusters but they are both backed by the same S3 and PG.
And then I can use DNS/LB trickery to swing traffic between the two.
Didip Kerabat
11/17/2022, 5:04 PMI am investigating absolutely zero downtime for Druid cluster at the moment…
c
Cory Johannsen
11/17/2022, 5:05 PMOh fascinating
d
Didip Kerabat
11/17/2022, 5:06 PMIf Cluster A’s Historical and MM use tierA and Cluster B’s Historical and MM use tierB…
I think it cloud work.
LMAO, could work.
r
Renato Santos
11/17/2022, 5:07 PMcloud work 😛
Renato Santos
11/17/2022, 5:07 PMwell, if you have they connected to the same ZK, they will be just one cluster still
d
Didip Kerabat
11/17/2022, 5:08 PMI think ZK would be non-issue, even if you have a dedicated Zookeeper per Kubernetes cluster.
r
Renato Santos
11/17/2022, 5:08 PMif you have separate ZK, each will think that they have a coordinator leader and may corrupt the database, but as long you only set up a historical, it's fine for a fast- recovery until you kill the other ZK/coordenators
d
Didip Kerabat
11/17/2022, 5:09 PMbecause internal traffic (brokered by ZK) never escapes to outside.
Didip Kerabat
11/17/2022, 5:09 PMand Historical segment files don’t contain cluster information, which is a great design! 👍
c
Cory Johannsen
11/17/2022, 5:10 PMWith the kubernetes extensions instead of ZK you can control which cluster owns the leader via kubernetes APIS.
Cory Johannsen
11/17/2022, 5:10 PMIf the ZK issue becomes problematic
r
Renato Santos
11/17/2022, 5:11 PMI think the main issue is to not ever have 2 coordenators as laeder running
Renato Santos
11/17/2022, 5:16 PMhaving a second 'dr' cluster with just historicals could indeed work, I think, if they use the same ZK and have a different tier in order to not cause each query routed on the 'active' cluster goes across a potential expensive network round-trip
and for the MM in the 'dr' cluster, connected to the same ZK, and add the task replica > 1 to have it HA but I don't remember if there's tiers for MM
but I think that's useless bc if the database or ZK fails on the active region, the dr MM would not be able to commit and serve queries when the task finsihes
d
Didip Kerabat
11/17/2022, 5:17 PMMM has category, which can be abused in similar manner 😛
Even better, it can be updated dynamically from the Overlord Dynamic Config 😄
Didip Kerabat
11/17/2022, 8:02 PMMM is less of a concern for me because the MM cluster can be down on the DR cluster.
Historical is much more important because data sync takes super long time.
g
Gian Merlino
11/21/2022, 10:04 AMIt's not that crazy 🙂
Gian Merlino
11/21/2022, 10:04 AMI've seen people do something similar for blue/green deploys: launch a whole new tier and then terminate the old one
❤️ 1
Gian Merlino
11/21/2022, 10:05 AMJust keep in mind that the clusters will be linked in some ways and so you still have a chance of correlated failure (for example if the shared metadata store or shared deep storage has problems)
4 Views