I am interested to learn what people have been doing for encrypting data and backing up data for disaster recovery. • I learned that S3 can be used as deep store where data can be stored encrypted. • Are there provided solution to encrypt data on disk and in transit? • How often is deep store updated. I am wondering if there can be data loss if cluster goes down and the data has not been written to deep store yet and has only been written to the local disk. Is that true? • Can the deep store segment be stored as backup, copying it to a different location. How can we consume it later?

Yes, that’s correct

Sorry I updated my question now, as I accidentally posted early. It has few questions.

• If you are using EBS, then there’s encryption there, and SSL for transit. • Every data that is ingested is always persisted in deepstore. Only callout is the consuming segments in realtime, if cluster goes down before segment is persisted, then consumption will start from previously saved checkpoint (ie no data loss). • As long as you have persistence in deep store and snapshot ZK (for metadata), you can always recover the cluster.

• If you are using EBS, then there’s encryption there, and SSL for transit.

EBS for deep store or the disk storage? I understood that the disk storage will not be encrypted as it adds to the query latency.

EBS for local storage. As I understand, there is some level of encryption (as in if someone got access to that physical disk they won’t be able to read). Pinot sees data as unencrypted. Check cloud provider for details

Thanks! I will explore that.