This message was deleted.

We've moved over to using containerised hadoop as deep storage as we found S3 performance pretty poor at scale (druid uses a single bucket to store all segments and delete/list performance at scale wasn't good enough to keep up). We wrote a very custom hdfs deployment to run as a stateful set in kubernetes and we've had no issues to date

Hi @Jaswanth Pinnepu Druid Operator does support hdfs configurations, pls check here and here.

Thanks for reply Nick and Adheip. Nick, May I know how much of an engineering effort did it take to setup the hdfs on k8s. I have read somewhere that there are a lot of pain points of setting up hdfs in k8s?

If you've got experience with hadoop outside of kubernetes (and some kubernetes experience), it's a relatively straightforward exercise to migrate the deployment.