This message was deleted.

Imply Clarity is not open source, it is a software that Imply commercializes that makes use of Druid metrics for multi-cluster monitoring and is also used to provide monitor views in the Polaris SaaS solution. KUDRAS ? If you mean the kubernetes-overlord-extensions for elastic task execution, it is part of the open source project and being actively tested and improved. We are expecting it to evolve from experimental status quickly.

Thanks for the info on Imply Clarity. Can you also share info on KUDRAS whether its opensouse or commercial? KUDRAS (*KU*bernetes *DR*uid *A*uto *S*cale) => https://imply.io/videos/introducing-kudras-kubernetes-druid-autoscaler-for-maximum-resource-utilization-and-speed/

I wasn't very clear, sorry. KUDRAS is intended to provide elasticity of task slot capacity. I am not sure about work being done on this in the community. The kubernetes-overlord-extension also provides task elasticity and is being actively worked on. This is the route that the project is taking to solve this need.

Thanks @Sergio Ferragut. As i understood from the link https://imply.io/videos/introducing-kudras-kubernetes-druid-autoscaler-for-maximum-resource-utilization-and-speed/ that that KUDRAS is used for Scaling all the Processes like Overlord, Coordinator, Historical, MiddleManager, Broker, Router but what you are pointing to is "kubernetes-overlord-extension" . From my view point they are different altogether. Please correct me if i am wrong

@Gururaj K.P I reviewed the video again to make sure. My understanding is that KUDRAS is a mechanism to autoscale MMs specifically. It was developed by Tikal and lead by their architect @Yoav Nordmann for their implementation. AFAICT, KUDRAS' autoscaling of MMs has the same objective as the "kubernetes-overlord-extension": making task execution elastic and responsive to the active workload. In the video Yoav mentions open sourcing their work but I am not aware if that has occurred. "kubernetes-overlord-extension" is being tested and used significantly by the community, so I think it will become the standard for auto-scaling tasks. @Yoav Nordmann please chime in, I'd be very interested in your thoughts on this.

Thanks @Sergio Ferragut, yes, KUDRAS is basically a precursor to the 'kubernetes-overlord-extension', which was not available at the time we developed it. In the meantime I am not working the that client anymore, but I know it is in production and working really nice. It was always out intention to open source the project, and i will ask them if this might be possible in the meantime. Still, i think it might be better to use the imply backed 'kubernetes-overlord-extension', as it seems this will be a standard plugin used by many. @Gururaj K.P, if you have further questions, please contact me!

Hi @Yoav Nordmann @Sergio Ferragut: Sincere apologies for the delay in reply. Thanks a lot for taking time and answering my queries. Also, Can you please let me know whether we can scale the DRUID processors (Overlord, Coordinator, MiddleManager, Historical, Broker, Router) using KEDA (Kubernetes Event-driven Autoscaling) and PROMETHEUS setup? if AutoScaling of DRUID is possible with KEDA, then are there any docs which can help ? or is it better to go with 'kubernetes-overlord-extension'?

I am not familiar with KEDA, but in general, elastic execution of tasks in Druid is going down the

kubernetes-overlord-extension

route.

@Sergio Ferragut: ok . Thanks for the information. So, using the 'kubernetes-overlord-extension' we can scale all the DRUID Processes (Overlord, Coordinator, MiddleManager, Historical, Broker, Router)?

kubernetes-overlord-extension

automatically create Peon pods that execute tasks, it creates them on demand. This is in lieu of Middle Managers, its also referred to as Middle-Manager-less configuration. The overlord and coordinator use a single leader instance to manage the cluster. There are usually two or three of these for high availability but they do not normally scale out, they scale up to accommodate larger clusters . Historicals are normally scaled manually (i.e. increasing the replica count in k8s). Historicals take some time to "warm up" as they must download segment data from Deep Storage to their local storage. Increasing or decreasing the number of historicals will cause the coordinator to rebalance the segments and their replicas across historicals. So this layer isn't normally auto-scaled. I think Brokers/Routers are stateless and could be configured for autoscaling on kubernetes, their workload depends on the number of concurrent queries being processed. Perhaps others in the community can share their experience on this front.

Hi Sergio, this is exactly what i was looking for. Thanks a lot for your inputs and information. This information will help us a lot

By the way, i was looking in the past into KEDA and using it for this purpose, but in the end we wrote KUDRAS, since extending KEDA seemed more tedious. Just as @Sergio Ferragut pointed out,

kubernetes-overlord-extension

is the way to go!

@Sergio Ferragut does ``kubernetes-overlord-extension` work for the EKS? is it different then the builtin autoscaler of druid that says it works only on ec2 and GCE. thanks

Yes. EKS should be fine. It is different than autoscaler. I'm not an expert on this but my understanding is that the extension spawns Peon pods as needed using kubernetes jobs. @churro can probably explain this better.