I have a usecase to plan setting up independent pinot clusters as disaster recovery in active/standby mode. To make sure that both the clusters have all the data when they are active, I am thinking of two approaches: 1. Send the kafka messages to both clusters so that data is ingested in both clusters. 2. Periodically transfer the segments from deepstore of the active cluster to standby cluster and upload the same. What steps are recommended if we want to go by second option: Is it OK to upload the segments from the active cluster (realtime_table) to standby cluster in offline_table. I think that uploading in realtime table leads to some problem and sometimes segment missing issues - probably due to offsets not following the same sequence in independent clusters. Any guidance is appreciated! Thankyou

1 is better if you want to reduce downtime. 2 is better if you can sacrifice availability for cost.

Thanks Mayank... some amount of availability is manageable for our usecase. If we go with option 2, I can copy the segments from deepstore from the active cluster and regularly keep uploading in standby cluster. I am wondering if we specifically upload this to offline table. Any guidance on achieving this so that I don't mess up the cluster state with respect to segments state and offsets within the segments.

You can load it to offline or real-time table, both are allowed. Note, if you are doing an hourly upload, then you have at least 1 hr of data not available during disaster. Also it depends on how quickly the new cluster can be resurrected (if servers are scaled down). So ultimately, it is cost vs availability tradeoff. You can choose based on the business requirements

Thanks Mayank... We were hitting into some segment missing exception post the upload of segments from the active site cluster to standby site cluster. We assumed that's due to segment/offset mix-up for realtime segments where may be sometimes empty segments were being created in deepstore even though ingestion was not happening in the standby site. Sometimes that may be due to some lag in kafka messages ingestion as well, so need to debug this better before seeking help around that. I wanted to get an idea if it makes sense to rather upload the segments to offline table instead of realtime table, as offset for segments uploaded from another cluster vs segments/offsets generated in the local cluster and consuming segments may conflict and cause issues for the segments metadata management and queries. I will get some tests done properly for both the options and share results/queries. Thanks for the positive guidance.

Check the state of those missing segments in Ideal state and external view