In Pinot deep store with s3 can we configure it in such a way that after the retention time in completed the data is deleted from pinot servers but not from s3 bucket?

The purpose of retention is to delete the data, what’s your use case that requires this?

The purpose is that the data will remain in the s3 buckets so that in future(after the retention period is over) if there is a need that data can be fetched and used. Here pinot servers will have the data according to the retention policy so pinot will only be able to query on that data only.

Then you can increase the retention to however long you want to keep the data. You should always add explicit time filter in your queries to only pick the data time frame you are interested in.

@Mayank so does this mean- once the retention period is reached, the data from deep store (s3 for example) is also deleted? I was of the assumption that once retention period is reached, segments from pinot server is deleted but it remains in s3.

Not sure if this helps, but an idea would be to just replicate the objects to another bucket where they're never deleted

Yes, retention will remove from configured deepstore, and you can always copy out. However, when you want to query again those segments would need to be pushed back. So cleanest way is to keep them around

There is a config for deleted segments retention. You could set that to something much larger than regular retention

@Neha Pawar so setting up that parameter will delete the segment from server but retains it in deep store? Could you please point me to the document if thats okay?