as i’m getting more comfortable with pinot (and getting more and more impressed, great work!), one thing i can see that would be really amazing would be a minion task that can snapshot the helix zookeeper state to deep storage. the automatic deep storage segment management is awesome, but ZK dies or gets corrupted, then you basically have to start from scratch again, especially over time. has anyone attempted to solve this? cursory internet searching hasn’t yielded me any generic ZK backup solutions

Zookeeper already has the ability to create snapshot and also the saves the commit log. You can always restore the cluster to the original state using the snapshot and the transaction logs

thanks, that was my only idea. maybe i can create a sidecar that periodically uploads snapshots to s3

are you running this on k8s?

yes, self-hosted k8s on bare metal

yeah, periodically moving the ZK snapshot will be good. Suggest mounting the data dir for ZK on a PV instead of local

yep, it’s on a PV but that’s just an LVM volume since it’s bare metal. should be easy enough to read snapshots from that volume and upload them to s3 with a sidecar container