or another question, is there a TTL for data in a ...
# generalt
Tim Chan
09/10/2020, 9:19 PMor another question, is there a TTL for data in a table?
n
Neha Pawar
09/10/2020, 9:24 PM
you can configure retention for the table. segments will be deleted once they become older than the retention
t
Tim Chan
09/10/2020, 9:26 PMotherwise segments just remain on the pinot server? and deep store is used for redundancy?
x
Xiang Fu
09/10/2020, 9:29 PM
Yes
n
Neha Pawar
09/10/2020, 9:30 PM
More like permanent storage. Servers can come and go. say you lose some servers, or add new servers, or want to replace certain servers. . The new servers will get their share of segments from the deep store.
Neha Pawar
09/10/2020, 9:30 PM
if you set retention according to your application needs, they will be deleted from both, servers and deep dtorage
t
Tim Chan
09/10/2020, 9:32 PMi think i understand better now. thank you both @Neha Pawar and @Xiang Fu
Tim Chan
09/10/2020, 9:34 PMso in summary, as my data grows (and i have set infinite retention) then my storage needs will need to grow accordingly (either add more nodes and/or resize pvc).
n
Neha Pawar
09/10/2020, 9:44 PM
Yes.
Btw, we have started working on Tiered storage: https://docs.pinot.apache.org/operators/operating-pinot/tiered-storage
Might be useful for you, since you mention infinite retention.
Currently only Pinot servers can act as storage tiers. But we plan to add deep store as a tier. Once that is done, even your server’s segments can live on deep store
t
Tim Chan
09/10/2020, 9:46 PMthank you. this is great!