is the deep store intended purely for backup/recov...
# general
b
is the deep store intended purely for backup/recovery? specifically i'm wondering: • is there any concept of cold storage where data is only persisted in the deep store and read as needed for queries? • the docs say the deep store doesn't typically (?) store indexes, and that zk stores segment metadata. are indexes stored in zk? or they're stored locally and rebuilt on restore?
m
Yes in OSS Pinot, deep store is for durability. • Concept of cold storage is there in OSS, but between SSD vs HDD : https://docs.pinot.apache.org/operators/operating-pinot/tiered-storage?q=tiered+stor • ZK only stores metadata. Deep store may or may not store indexes (depending on how the segments were created before pushing to deepstore - it won’t explicitly remove an index). Servers are capable to dynamically rebuild indexes as needed.
If you are looking for deep-store as tiered storage then please check out: https://www.startree.ai/blog/introducing-tiered-storage