Hi All, I am trying to understand the use of buffer size (

druid.processing.buffer.sizeBytes

) and merge buffers (

druid.processing.numMergeBuffers

) in the broker. Configuration: https://druid.apache.org/docs/27.0.0/configuration/ For buffer size (

druid.processing.buffer.sizeBytes

), the documentation has the same content for both historical & broker:

This specifies a buffer size (less than 2GiB) for the storage of intermediate results. The computation engine in both the Historical and Realtime processes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed. Human-readable format is supported.

Does this mean that the buffer size doesn't matter to the broker? Cluster tuning: https://druid.apache.org/docs/27.0.0/operations/basic-cluster-tuning#broker In cluster tuning documentation, there is the below guidance on the direct memory sizing of the broker.

On the Broker, the amount of direct memory needed depends on how many merge buffers (used for merging GroupBys) are configured. The Broker d*oes not generally need processing threads or processing buffers, as query results are merged on-heap* in the HTTP connection threads instead.

So where does the broker use the merge buffers if the query results are merged on the heap? Is it only in cases of nested groupBy? In our usecase, almost all the queries are "groupBy" queries so we are looking to increase the number of merge buffers and processing threads on the historical processes. On the historical, is it correct to say each processing thread will use atleast one merge buffer for a "groupBy" query? (

druid.processing.numThreads

>=

druid.processing.numMergeBuffers

) Along with the above change, we believe that the number of merge buffers on the broker also needs to be increased to same value or higher than that of the historical. As this will affect the direct memory requirement on both the historical & broker processes, we wanted to understand how the buffer is actually used on the broker. Can the broker have a smaller buffer size than the historical? Thanks, AR.