Hello All, out team is trying to migrate Pinot wit...
# general
h
Hello All, out team is trying to migrate Pinot with real-time tables to a new cluster in AWS ECS, is there any document about data migration, I didn't find any docs online.
g
We were in the same boat and migrated data from the old cluster to a new cluster using the following method, Note: We only have real-time tables but the following method works for offline tables too 1) Pause real-time ingestion on all tables in old cluster 2) Make sure all consuming segments are committed and upload to deepstore in old cluster 3) Create same tables on the new cluster with same Schema and configuration 4) Upload segments from S3 bucket (deepstore) to new cluster (using following API through some scripting), note that the new cluster will need access to S3 bucket used in your old cluster
Copy code
curl -k -H "UPLOAD_TYPE: URI" -H "DOWNLOAD_URI: <s3://path/to/data.tar.gz>" -X POST "https://<controller ip>:9000/v2/segments?tableName=<tableName>&tableType=<tableType>"