Hi Team, We are using HDFS as deep store. How do ...
# troubleshooting
a
Hi Team, We are using HDFS as deep store. How do we migrate to different HDFS cluster with 0 downtime? Pinot version 1.0
m
There isn't an out of the box tool for this afaik. You can build a new cluster/table in parallel (so data will be duplicate for some time), and then migrate your app live by redirecting traffic to new cluster.
x
is it for HDFS migration? then just copy over data from old to new hdfs with same path and switch the hdfs conf for all controller/server/minions
a
So we move the folder /hdfs/comtroller/ to new cluster and switch hdfs conf files?
m
No, you need a brand new table. Segment ZK Metadata stores the segment url in deepstore
a
If we keep the path exactly same, will it work?
m
should work, but test it out
a
Sure , will try
Hey @Xiang Fu @Mayank, So incase of creating whole new Pinot Cluster, how to restore data for _offline and _realtime table from old Pinot cluster ?
x
backup and restore zookeeper
in new cluster, tag all the old instances as
untagged
then rebalance the entire cluster if deepstore data are already there
a
thanks @Xiang Fu. Wanted your advice on one more approach. We need to shift the entire cluster to another DC. ( Austin > north virginia ). So is it advisable to, add pinot compenents which is hosted in NV into current cluster which is in ADC? Summarizing two approaches till now: 1. Setup New Cluster in NV DC -> backup and restore zookeeper -> keep deep store data ready in NV HDFS -> rebalance 2. Add NV components in existing Cluster -> switch the cluster to NV HDFS -> Rebalance -> Remove older components.