I'm trying to better understand the resources used by queries on historicals so that I can tune / query lane as needed. I've read the docs but I have questions on how historicals process each segment in parallel and how that might consume CPU resources on the node. For example, if the historical has 100 segments to query will it end up utilizing all available threads (and potentially max out the CPU ) as configured in

druid.processing.numThreads

or is there a limit? Will any other queries be able to run if one query (or a small handful of queries) is utilizing all

numThreads

or are there more threads actually executing queries (e.g. up to

druid.server.http.numThreads

)?

Historicals use 1 processing thread per segment If you’re having a query which is tying up processing threads with long

query/segment/time

per query, you might want to change the priority of those queries so that other queries can jump the queue From https://druid.apache.org/docs/latest/querying/query-context#general-parameters

Query priority. Queries with higher priority get precedence for computational resources.

If I had a large query where aggregation took 10 seconds; do all other queries wait in a queue for 10 seconds for the first aggregation to complete?

The thread used for aggregation is not the one that’s processing the segments. The sorting/merging happens on the queries http thread afaik, so queries don’t hog up the threads when doing merging or sorting. The number of merge buffers for aggregation is limited however, so queries can be blocked on that

We notice CPU as the bottleneck so my gut says the threads are fully utilized (and not blocked on merge buffers) but I'll double check our configs on that. According to documentation (assuming a group by), the aggregation is done on the historical and uses all available threads (it's supposed to be quick I'm using 10 seconds as an arbitrary example).