Hi <@UDT7GFEG6> <@UDQU92KBK> <@U03NQ2WUUQ5> - Can...
# generalp
Pallavi Kumari
06/27/2023, 5:26 AMHi @Jackie @Mayank @Mark Needham - Can you help with this query :
We have setup Pinot in two clouds in production and suppose both clouds are down how do we follow DR strategy in Pinot for realtime tables because we will have Zookeeper/Controller /Broker/Server nothing working for that duration.
Can we have cross DC replication of data in Pinot?
m
Mayank
06/27/2023, 5:27 AM
By two clouds you mean two regions, or two vendors?
p
Pallavi Kumari
06/27/2023, 5:27 AMTwo regions
m
Mayank
06/27/2023, 5:28 AM
If both regions are down, Pinot will be unavailable, like any other service right?
Mayank
06/27/2023, 5:31 AM
But if you mean how to resurrect after the regions come up, you need to ensure that ZK is snapshotted as well as you are using DeepStore (for data). Once the region comes back up, ZK will need to be restored from the last saved snapshot, and assuming data is available in deep-store, that is all you need, and the cluster will come back up from that state.
p
Pallavi Kumari
06/27/2023, 5:37 AMIf both regions are down, Pinot will be unavailable, like any other service right?Correct. So, this is the approach thats there in our mind. Our Pinot setup includes Azure blob storage as deep store. If we bring up another region and do the necessary settings, it will be pulling all the data from deep store and point to new server nodes if we change the table tenant to bring the cluster up and running again
Pallavi Kumari
06/30/2023, 3:16 PMHey @Mayank How do I disable deep storage in Pinot using config?
m
Mark Needham
06/30/2023, 3:17 PMI don’t think you can disable it. If you don’t specify it’ll use the controller
p
Pallavi Kumari
06/30/2023, 3:18 PMI tried not passing any parameter related to deep store configs. But My controller is not coming up. Sharing my configs :
m
Mayank
06/30/2023, 3:20 PM
Controller data dir should point to something that is not deepstore but shared across multiple controllers (eg NFS). Curious, why do you want to disable it?
p
Pallavi Kumari
06/30/2023, 3:21 PM Copy code
controller.helix.cluster.name=<my cluster name>
controller.port=9000
controller.vip.port=9000
controller.local.temp.dir=/mnt/pinot-data/controller/data
controller.data.dir=
controller.zk.str=<my zk>
pinot.set.instance.id.to.hostname=true
controller.enable.split.commit=true
controller.disable.ingestion.groovy=falsePallavi Kumari
06/30/2023, 3:21 PMOk trying after passing the NFS