Hey folks. We're hosting a druid service set up using helm on k8s. If folks are also hosting similarly, I wanted to enquire about your historical load up times? When we do a historical restart the entire segment collection is downloaded on to disk, but since we're using a persistent storage the data from the previous druid instance is already present there. So can we not just use that data? Or are we doing something wrong which is causing the high historical load ups? Is it to do with the fact that when we restart the historical, the coordinator is also restarted. Apologies for repost.

It should just reuse the data that is already there. It is usual to use stateful sets for the historicals so that each historical knows which volume its data was on.

That's exactly what we were expecting but we're seeing the re download of segments everytime the historical restarts

Then it is likely there is some subtle k8s misconfiguration. We use this setup and it works well. It would be pretty bad for us if we had to redownload all our data fresh on every restart. In fact it would be impossible to rolling restart the historicals as each would take too long to become responsive.

Understood. Thank you @David McHealy. I'll have a deep dive soon so I'll keep this in mind

Another thing that caught me when I first experimented is I thought I was writing data to a mounted volume, but was actually writing to the pod's ephemeral storage, basically, so it was just lost at restart. Obviously under a real workload you'd run out of disk space, but just messing around you might not notice.