Hi all, We are fine tuning few druid properties an...
# general
a
Hi all, We are fine tuning few druid properties and had few doubts when going through docs, could someone please clarify 1. Query processing buffers/threads •
druid.processing.buffer.sizeBytes
•
druid.processing.numMergeBuffers
•
druid.processing.numThreads
I want to understand how are these used when processing the druid queries, couldn't get much clarity from docs. Could anyone please explain these? I see there are formulas in the docs, but wanted to understand the logic behind them. For example, when we increase
numThreads
, more query processing and row scanning happens in parallel, but at some point groupBy/aggregation phases require merge buffers. If merge buffers are insufficient, does query processing block waiting for a merge buffer? What are the key factors to decide appropriate values for these? https://druid.apache.org/docs/latest/configuration/#historical 2. Heap vs direct memory allocation Here, https://druid.apache.org/docs/30.0.1/configuration/#segmentwriteoutmediumfactory it mentions that for MiddleManagers, heap and direct memory should be the same size. Why is that requirement there? Also, does this recommendation apply to Historicals as well? Thanks