https://pinot.apache.org/ logo
#general
Title
# general
l

Liran Brimer

07/08/2021, 12:55 PM
Hi everyone, we are evaluating Pinot and one of our requirements is to be able to encrypt our client's data on the disk (in memory it can be decrypted). is such a thing possible? and if so, we may also need to encrypt it with a different encryption key per client (each client's data would be encrypted with a unique key dedicated to that client). is there a way to achieve that? thank you so much
m

Mayank

07/08/2021, 1:41 PM
Pinot does support encryption of data copy on deepstore. However, the local server copies on disk need to be decrypted to maintain low latency. The per client encryption requirement is an interesting that I came across in the past and opened an issue to track https://github.com/apache/incubator-pinot/issues/7138
l

Liran Brimer

07/08/2021, 1:50 PM
It would be a great feature! TBH i’m not sure what you mean by “deepstore”. is that that storage consumed by Pinot, or the source of data owned by us and ingested into Pinot?
m

Mayank

07/08/2021, 2:02 PM
Pinot uses deep store to maintain a golden copy of the data ingested. It supports deep stores like S3/ADLS/GCP/etc. That copy can be encrypted.
Pinot servers store a copy of the data on local disk for faster serving (today), that copy does not support encryption.
l

Liran Brimer

07/08/2021, 3:08 PM
thanks for the explanation. and those are copies of all the data, or just subsets of it ?
because i’m trying to understand whats the actual value of deep-store encryption, if the same data is decrypted on other disks?
m

Mayank

07/09/2021, 2:53 PM
Each server will copy only the shards it needs to host. The encryption on deepstore was done for a specific feature ask we had at LinkedIn for ADLS as deep store
v

Vaibhav Mittal

07/12/2021, 5:01 PM
Is there a plan to support data encryption on the local disk as well?
m

Mayank

07/12/2021, 5:19 PM
@User No concrete plan at the moment.