Hi All, We are exploring Pinot. And have few quest...
# general
m
Hi All, We are exploring Pinot. And have few questions, hope this is a right group to put across if not kindly let me know the right group. ( Question ). Can pinot work with deep storage, we are looking for inverted index and columnar which pretty much solved in pinot servers itself. Should we need deep storage? like hdfs? (Answer we find in ) https://docs.pinot.apache.org/release-0.4.0/developers/tutorials/store-data . Yes you need hdfs or s3 to have persistence. (Overall Question is ) : Does it mean one need hdfs or s3 for persistent storage and you can't make pinot server itself as a persistent layer as incase of ALL pinot server shutdown and started, data will be gone? (edited)
m
Yes deepstore is needed for data durability
k
Just to clarify.. Pinot server is shutdown and restarted, the data will still be on local servers.. deep store is for backup when the server gets deleted and you want to bring a new replacement
m
@Kishore G does pinot server has replication factor in it? So the data gets distributed within pinot server itself so incase a pinot server is gone down the replica pinot will form new data of failed node ?
m
Yes, you can specify desired replication and Pinot will replicate the data accordingly across different servers.
m
Got it, this answers pinot works without deep storage playing with its replications… that avoids deep storage requirement for me.. @Mayank if pinot server documentation can fill this would be helpful ..
m
Replication is for availability, and deepstore is for durability. You need both for production system. Whats your concern with deepstore? Note something like nfs can also be deepstore
m
If am able to recover when a node is crashed with another node due to replica , i dont need durability factor.. and with deep storage it comes with cost as well..
m
What if all nodes crash? Deepstore is not optional in Pinot, you need to have controller point to a deepstore it can either be s3, hdfs, adls or even nfs
m
in that concept what if all hdfs crashes? Sorry didn’t want to be argumentative, am trying to find what best fits between if hdfs is a must have or good to have. If its only durability we are okay to let go hdfs due to replication factor support and taking a chance with probability of more than one node crashing at same time ..
k
Mannoj, you can definitely run it without deep store as long as you are aware of the cases where you might lose data. when you run without the deep store, you can also turn on peer download mode and the servers can download from peers if they dont have the data