https://pinot.apache.org/ logo
m

Matt

02/02/2021, 8:57 PM
Hello is there a way to backup and recover all segments? In situations like, server is not recovering and you are forced to recreate the table but don’t want the indexing to happen on previously indexed data but resume from the crash point.
k

Kishore G

02/02/2021, 10:44 PM
Segments are always backed up in the segment store automatically
m

Matt

02/02/2021, 10:48 PM
Is it possible to set Efs/nfs as segment store. It will be very helpful if you could share some docs related to this.
k

Kishore G

02/02/2021, 10:50 PM
yes. you can have NFS as segment store. @Mayank do you have a diagram on how it works at LinkedIn
m

Mayank

02/02/2021, 10:51 PM
Yes NFS can be one of the deep-stores.
IIRC it is the default PinotFs implementation. So simply setting the controller dataDir to point to a NFS mount point should set you up @Matt
m

Matt

02/02/2021, 11:08 PM
Thanks @Mayank, I tried that but I cannot see all the segments coming to controller folder. Controller segment folder size is very less compared to server data folder
The file colums.psf is missing from all the segments in the controller data folder
So controller is only saving segment index without data . I am guessing there could be some configuration setting somewhere that enables the controller to store the data as well?
Actually it is there, sorry my bad
m

Mayank

02/02/2021, 11:40 PM
So it works?
m

Matt

02/02/2021, 11:41 PM
All segments are in controller data folder zipped. But I don't know how to fetch it back to pinot server when required.
Wondering reload all segment api will pull segments from controller if it is missing from the server ?
m

Mayank

02/02/2021, 11:46 PM
Yes
m

Matt

02/03/2021, 12:24 AM
Thanks will try it
m

Mayank

02/03/2021, 12:25 AM
Yes, loading of data from deep-store/NFS to servers is transparent to users (you should not need to worry about that).
m

Matt

02/03/2021, 12:26 AM
Great design there, thanks for sharing
👍 2