https://linen.dev logo
Join Slack
Powered by
We are having issues with certain slow queries ess...
# general
j
We are having issues with certain slow queries essentially starving all other queries. We have tried query laning (using 200 segments as the threshold) to some success but it isn't perfect. We are now running into an issue where that 200 segment threshold works for some datasources, but definitely not others. For example, we would like it to be 100 segments for some datasources and 500 segments for others. Is there any way to prioritize traffic based on some other performance metric (e.g. the # of rows or # of bytes)??? The cluster runs fine most of the time but every once in a while, some queries come along starving out all of our other queries, and then we get endless user complaints that Druid is unavailable / unreliable / etc...
m
We had this same problem.
Our solution is by building something like https://github.com/apache/druid/pull/18148
We noticed that these slow queries are slow in the per segment processing
(you can verify this with the per segment query time metric)
as such, we are able to fail fast these slow queries using the above feature
instead of waiting for the query timeout period to elasped
j
Yes, I noticed as well that there's a lot of times spent in segment scanning (cpu-wise).
What will this do to the query? Is there automatic retry? Ideally I want these queries to continue to execute but without starving every other query. We tried hilo query laning for a while but had to revert it due to it failing the queries. We currently use threshold prioritization, which helps, but it doesn't solve all of our problems.
m
This will fail the query
the idea is that these expensive query will eventually timeout anyway
or even if they don’t, they are too expensive to be running on Druid
Instead of having these expensive queries hogging up processing threads for a long time, we fail fast to free up resource
j
I don't mind having a few expensive queries running. It is necessary for many of our OLAP use cases. They do not time out on our system but they DO starve queries that should take milliseconds into taking > 10 seconds which severely degrades user experience. Frankly, if it's too expensive to be running on Druid then it's too expensive to run anywhere. Nowhere else have I been able to crunch billions of rows in 1-2 minutes. That said I will look into this feature. Possibly make a PR to allow retry or pausing. If I can interlace cheaper queries with more expensive queries then I'm golden.
m
I don’t mind having a few expensive queries running. It is necessary for many of our OLAP use cases. They do not time out on our system but they DO starve queries that should take milliseconds into taking > 10 seconds which severely degrades user experience.
I see. this is a different to the issue we had. Our expensive queries are really expensive and usually is due to user trying out a new query. Thus, we are fine with one user having failed queries instead of all of our users having slow queries
Frankly, if it’s too expensive to be running on Druid then it’s too expensive to run anywhere. Nowhere else have I been able to crunch billions of rows in 1-2 minutes.
Another option could be running on MSQE instead. Sometimes we also found that these queries does haven’t to be run in 1-2 minutes and our users are better off using Spark haha
j
Spark is another option for those 😄
Correct me if I'm wrong but Druid will already pre-empt those threads on query timeout? If not then I could at least enable this feature using the same timeout as our default query timeout...
m
Druid will pre-empt those threads on query timeout. But usually query timeout is much longer (i.e. order of minutes), while per segment timeout can be configured much short (order of seconds). Hence, a query can fail faster and release resource (threads) faster.
✅ 1
Open in Slack
PreviousNext
Powered by