Hello everyone, we're currently working on a POC a...
# general
o
Hello everyone, we're currently working on a POC and had a question around deepstore and table replication: we have two separate pinot instances, one of them is receiving live data into the tables. We would like to copy segments from the live pinot instance to the other instance so that we can recreate tables. What's the best way to go about this given zookeeper still maintains some metadata, so the deepcopy, according to docs, would not be enough for recreating the tables in the other pinot instance?
It doesn't seem like there is a direct way, according to the faq, so wondering what the best way to do this is?
m
If I may ask, what's the reason for moving data from one table to another?
o
We have deployment-specific instances that are isolated and collect data per deployment. We then want to port the data over into our centralized "watchtower" instance that has data from all of our deployments
m
To answer your original question, you can write some automation that uploads data from S3 of first instance into the second table (using rest-api). However, from what you are describing, it might make more sense to evaluate how you are using Pinot (esp why is there a need to isolate data per deployment)
Sounds like you might be retro-fitting Pinot into an existing architecture?
o
The deployments are customer-based and need to be isolated due to compliance, so it's unlikely that we'll move away from that model of per-deployment instances
got it, so it will be data imports from object store into tables. Just to make sure I understand correctly, with that options it's not so much copying/porting the segments/tables, but rather the underlying data and then importing it back into offline tables. Is that correct?
m
By Pinot instance you mean the entire Pinot cluster?
o
Yes, separate clusters per deployment
Yes, separate clusters per deployment
That is too much of an overhead. A single Pinot cluster can support thousands of tables, all of which can be isolated by using different tables (logical), or even tenants (physical)