This message was deleted.

on historical:

druid.server.http.numThreads=2000
druid.processing.numThreads=31

we’ve been seeing this on larger clusters but can reproduce the behavior in a cluster with only 4 historical boxes.

Do you know what the cpu utilisation is on the historicals when this issue occurs?

For historical CPU was underutilized most of the time and no spike during the period. eg. < 10% for cpu.sys.pct/cpu.user.pct

The top level query is a union query?

that kind of make sense, but still can explain for the first subquery it fire 16 secs later after receive the top level query

what is the CPU utilization on the broker looking like? Also what is your qps? If the union queries have multiple queries then one connection is open per sub query to the historical.

I suspect the issue is on the broker side though since it took 16 seconds to launch the query. You can track : jetty/numOpenConnections on the broker. https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java#L540 The newer versions in druid have more metrics around jetty . If the numOpenConnections is > druid.server.http.numThreads (of the broker) then you might be connection bottlenecked and see why the queries are taking so long to respond. Maybe the answer is external as in new q's on the system and we need more brokers to serve them.

@Karan Kumar thanks for the responses. we also suspect some thread/connection/queue bottlenecks there, so checking the numOpenConnections might be a good idea. Some questions - Within what range the number is proper/acceptable? What config control that number? what happen to those requests that cannot get open connection? Regarding the idea of adding more broker, we haven’t give it some thoughts, as even when we tested with qps = 1 for 60 mins we can still see this issues started to happen after 10 - 20 mins

Some questions - Within what range the number is proper/acceptable? What config control that number?

I think if

numOpenConnections

are constantly less than <

druid.server.http.numThreads

we should be good.

what happen to those requests that cannot get open connection?

Idk I have not tried it. I think the connection might be rejected but that would be more of a jetty thing.

Regarding the idea of adding more broker, we haven’t give it some thoughts, as even when we tested with qps = 1 for 60 mins we can still see this issues started to happen after 10 - 20 min

Since your query is a group by/topn query you might be using merge buffers which might become a bottleneck on the broker. You might want to increase the number of merge buffers and see if that helps.

druid.processing.numMergeBuffers

druid.broker.http.numConnections

I think this is more related to broker to historical/indexer. How many historicals/taskslots does queries generally get served by would help decide that number. So for one jetty thread, ie "numThread" we can go out and reach out to 10 historicals and 5 peons so would require atleast 15 connections