Hello 👋 druid community - I have a question about scaling brokers - when to do it, how to do it, and its impact on Historicals We are setting up a druid cluster to handle QPS of about 100-120. Below is the setup and configs Brokers Count: 2 | vCPUs: 32 | No Tiering

druid.server.http.numThreads=66

(MAX(10, (vCPUs * 17) / 16 + 2) + 30)

druid.broker.http.numConnections=45

Historical Count: 3 per tier | vCPUs: 64 | Tiers: 3

druid.server.http.numThreads=100

(MAX(10, (vCPUs * 17) / 16 + 2) + 30)

druid.processing.numThreads=63

(vCPUs - 1)

The numConnections on brokers is set such that each historical can handle all incoming connections from brokers (2 * 45). Also, we haven't setup broker tiering. When we test on loads much above 150 QPS, we are noticing increase in queue size - which I think is expected. However the CPU utilization on the broker and Historicals are less than 50%. This makes me think the resources are under utilized. Questions If we have to scale brokers for future higher query loads 1. Should we start by increasing broker numConnections and historical numThreads without changing the instance type? If we do this, we will deviate from the formula suggested and what could be the implications? 2. Should we add a new broker node? This forces us to vertically scale each Historicals to ensure that each node can accept all incoming broker connections. Is this a right way to think about this or is it okay for Historicals to have fewer server threads than all incoming connections? Considering the CPU utilization is low on Historicals, I think we should do (1) till we hit an optimal CPU utilization. I also feel (2) is very restrictive. Would be helpful to hear your thoughts. We also understand that making queries faster is also essential to improve QPS. We are also working on that but this question was specific to scaling brokers.

What type of queries are you running? If they're GroupBy, I would also recommend checking the mergeBuffers as that can cause limit the amount of parallelism even if there are threads being unused.

We do have a considerable number of groupBy queries. The

mergeBuffer/pendingRequests

metric doesn't seem to show up despite enabling the

QueryCountStatsMonitor

module. Are there other symptoms for mergeBuffer contention?

Regarding your original questions • You probably won't see much benefit by increasing numConnections or the historical processing.numThreads. • Adding another broker will probably help, you probably don't need to increase the http.numThreads on the historicals unless you see the

jetty/threadPool/queueSize

metric jump.

you probably don't need to increase the http.numThreads on the historicals unless you see the

jetty/threadPool/queueSize

metric jump.

This makes sense. Will experiment on this. So it looks like Thanks We are on v 30.0.0. We do see the other metrics published by

QueryCountStatsMonitor