Hi team. I have ran a cluster in pinot with deep_s...
# troubleshooting
r
Hi team. I have ran a cluster in pinot with deep_store. Segments were going in the S3 smoothly, the retention provided at that time was 7 days. the server crashed due to heavy load on server during festival. We decided to reduce the retention from 7 to 4 days. After doing that I reloaded all the segments. How long will it take the pinot to resume ingestion? Basically how long will it take to delete older segments? @Xiang Fu @Mayank
m
You didn’t need to reload segments. You can call retention explicitly via api
r
which API?
m
Check swagger for periodic task api
x
Usually controller retention manager should kick in within 5mins after you restart controller
r
It didn't start. So I had to re-ingest the data again
the server was crashed maybe thats why
server disk was 100% full
x
controller should remove data from zk and deep store
but if your server disk is full, you need to clean it up
r
yeah thats why I started the ingestion again
a
hi @Mayank @Xiang Fu Just few doubts on this thread. I see but what if I need the data from inception for my use case ? in that case I wont be able to setup a Data retention policy . Also deleting data from server ? Doesn’t server in Pinot queries data from deep store where the controller pushes it after realtime Kafka ingestion and then the server loads the data to its volume for querying from broker ?
m
Answered in dm. But answering here as well. Not specifying retention means infinite retention. Servers store local copy of data for fast serving, deepstore is for DR