This message was deleted.

it's kinda hard, there's a few discussions here at but I don't know if they are still up or if slack already removed them basically, druid is HA in the same DC if your data is on S3, and you consider S3 as external or already hyper HA, then that's "done", or have a way of replicating the segments ASAP druid (usually) uses ZooKeeper to keep track of it's peers, and it needs a central database (mysql or postgres) to keep track some information this database definitely should have it in HA, even if in the same region having "two" druid clusters in two DC, and joining the same ZK, will turn into a big cluster, and if you have the database set up as well to HA in another region, maybe that's already enough to handle most failures that can happen but it's a lot of moving pieces and you need to think about each one of them, eg: is the kafka also HA and have good network to all regions, when the MM task fails it re-read al data since the last offset saved in the database (the coordinator election also would had to be run) in any scenario, when an entire region where a elected druid-supervisor goes down, I expect at least ~ 3 minutes of downtime before druid resync everything with ZK and start new peons (tasks that the MM will start to sync the missing data in the other DC) and that is considering that the database (metastore) was not affect at all

Another option might be to simply load your data into two clusters, one in each DC.

Thanks for prompt response @Renato Santos & @Ben Krug. Yeah it does look complicated. Wonder if doing something simple as SAN replication would do the trick - so in essence 2 clusters - one in each DC with data replication done via SAN replication or cluster to cluster replication - if this function exists. Data ingress to both clusters and have them running independently in each DC

I think normally people don't use SAN much for low-latency distributed databases. Although I suppose it could be done, if response times are good.

I agree, I would not see SAN used here Maybe if you are using MinIO as a backend for the object-storage [segments], and use the rancher Longhorn for backup/distribute between multiple DC's But I think it's just easier to design your system to either accept that druid may have brief moments of unavailability, or consume the data twice in two kinda of 'split brain' scenario, but if you are using kafka as a source, you can have pretty much a guarantee that the data will be the same if you have good process when updating both druid ingestion spec at the same time

@Kieran Munt @Andy Lamb

Hi @simon mikolajek, I am with the DevRel team at Imply and close to your timezone. Please let me know if I can help you!