Hi everyone, we are evaluating Pinot and one of our requirements is to be able to encrypt our client's data on the disk (in memory it can be decrypted). is such a thing possible? and if so, we may also need to encrypt it with a different encryption key per client (each client's data would be encrypted with a unique key dedicated to that client). is there a way to achieve that? thank you so much

Pinot does support encryption of data copy on deepstore. However, the local server copies on disk need to be decrypted to maintain low latency. The per client encryption requirement is an interesting that I came across in the past and opened an issue to track https://github.com/apache/incubator-pinot/issues/7138

It would be a great feature! TBH i’m not sure what you mean by “deepstore”. is that that storage consumed by Pinot, or the source of data owned by us and ingested into Pinot?

Pinot uses deep store to maintain a golden copy of the data ingested. It supports deep stores like S3/ADLS/GCP/etc. That copy can be encrypted.

thanks for the explanation. and those are copies of all the data, or just subsets of it ?

Each server will copy only the shards it needs to host. The encryption on deepstore was done for a specific feature ask we had at LinkedIn for ADLS as deep store

Is there a plan to support data encryption on the local disk as well?

@User No concrete plan at the moment.