Hi guys, In cluster tuning guide. For historical node

A general rule-of-thumb for sizing the Historical heap is (0.5GiB * number of CPU cores), with an upper limit of ~24GiB.

why is num cpu * 0.5GiB? not processing thread * 0.5G? why does the upper limit set to 24G? what is the reason behind the number? My data node has 64 core, 512GiB. heap: 24G. , Cache is set to 2GB DirectMemory(60G) with processing thread(40), merge_buffer(10) , bufferSize(1G), after some query congestions, I see some of historicals are keep having high CPU (near 100%) and sees GC 30 times per minute.

• Processing threads is generally set to # vCPU - 1 ... so it's close to the same. Maybe one extra count for overhead? • Upper limit of 24GB ... not sure if that was related to an older Java version, but I do know if you keep the heap under 32GB you will gain benefits of more compact object pointers. But I am not an expert on JVM ... hopefully someone else will be able to provide better info.

Found the issue, it was not the GC. Some of historicals keep running hash join query. it never terminates. the query timeout was set to 10000ms. I believe this issue is related to https://github.com/apache/druid/pull/16773 But what I do not understand is why on specific node? data is distributed over 20 historiclas. Query returns fail but historicals were strill processing.