Hey community, Wanted to know if anyone is facing ...
# general
y
Hey community, Wanted to know if anyone is facing issues we face in our recent Druid efforts. We are using Druid For years (version 30 at the moment) and currently working on adding MSQ ingestion tasks. We have a very high load and we encounter that when there are too many tasks for the middle manager to handle, it creates only
query_controller
workers and no capacity is left for
query_workers
which means tasks arent being handled at all. The obvious solution is scaling the middle manager but we wanted to know if there is a way of giving priority or splitting the workloads. Did someone encounter this when using MSQ? Any suggestions or ideas?
k
There was a community PR to limit the number of task types in case such an issue is hit.
y
Do you have a link to it? does anyone knows if its rolled out yet?
k
D33
y
😭 Ever heard of a way to deal with it in Druid 30? Upgrading at the moment would be tons of work for us
k
I would strongly suggest to upgrade to the latest and the greatest version
There should not be any work for the application teams.
Druid is designed to be rolling upgradeable.
Other way is to create a fork, backport this patch. Pray it is easy to apply and then roll it out or use the d33 and roll it out.
y
We have a high scale Druid setup, with 100s of servers and 100s of TBs of data. our whole product depends on it. we usually reserve time every couple of years to upgrade the cluster but it takes us 1-2 month each time
g
with Druid 30, the best solution i can think of is to use the "Worker select strategy" feature with
strong: true
to assign controllers to a dedicated pool of MMs. this way, when those MMs fill up, controller tasks will queue up but workers can still launch (on the other MMs) you could also pull the patch @Karan Kumar mentioned back to Druid 30 so you don't have to do the major upgrade with every other change from 30 -> 33
s
We’ve encountered a similar issue when working with MSQ ingestion under high task loads, and there’s a suitable solution available. Starting from Apache Druid 30, a feature has been developed to control the balance between _*query_controller*_ and _*query_worker*_ tasks for MSQ tasks. This addresses the exact scenario you described - ensuring that middle managers don’t get overwhelmed by _query_controller_ processes, leaving no slots for _*query_worker tasks*_. This feature is not included in the official Druid 30 release, but we at Deep.BI have backported it into our custom patch release: druid v30.0.1-patch4. The patch includes all relevant changes to support this behavior, and the same feature is also available in all our builds based on Druid 30+. As detailed in our v30.0.1-patch2, the enhancement introduces a new taskLimits property in the select strategy, configurable via the Overlord dynamic configuration. It allows you to limit the number of tasks by type, running at the same time. This limit controls the percentage or absolute number of task slots that can be allocated to specific task types. When the limit is reached, tasks will wait for resources instead of blocking other tasks and failing due to timeouts. For instance, you can limit the number of query_controller tasks to at most 10% of the slots available. You can find more information on the feature here: https://druid.apache.org/docs/latest/configuration/#tasklimits