Hi Team, We have 2 brokers and 4 historicals in our cluster. The broker configuration is as below: Heap: 16GB DM: 22GB druid.broker.http.maxQueuedBytes=10Mib The brokers restart every week. By the 4th or 5th day, the used heap memory (jvm/mem/used) is > 15GB and both of them crash with OOM on the 5th day. The used heap memory is ~2GB on startup and increases steadily over the next few days. According to the documentation, broker uses heap to store: Segment timeline of all available segments Segment metadata cache of all available segments Partial unmerged query results The cluster has around 100k-120k available segments and tons more inactive segments. Our understanding is that inactive segment data is not cached on the broker and hence should not contribute to the memory usage. Most of the queries are groupBy queries which would need the results to be merged on the broker. But this memory should be released once the query results are returned to the client. Correct? The recommendation for broker heap sizing is 4-8GiB for a small/medium cluster (~15 servers or less). What could be taking up so much of heap space in the broker process? Could there be some memory leak which is leading to the OOM? Are there any monitors in Druid which would detect memory leaks? In addition most of the ingestion is done using MSQ. Not sure if it adds additional load on the broker heap but I do see the queries (INSERT/REPLACE) logged in the broker logs. Druid Version: 24.0.2 Thanks, AR.

• do you have the heap dump of the oom? that would help in triaging the issue • Group by queries release the query results once returned to the client • OOM shouldn’t happen, brokers should have sufficient guardrails to prevent against OOM • MSQ places almost no load on the brokers

Hi Laksh, We also reindex a lot of data (REPLACE OVERWRITE) on a daily basis. This is because some of the ref data changes frequently and we need to rewrite the full history (sometimes) for one or the other table as we cannot update specific columns in the existing data. Could this churn of segments lead to increase in broker heap size usage? I don't have a heap dump of the OOM. Unfortunately this only happens in PROD (data volume). So as a temp fix, we are planning on increasing the heap size to 20GB this week. I can add the jvm param to generate the heap dump if it crashes again. I wouldn't be able to share it on the forum. If there is something specific that can be checked in the dump, I can take a look. Thanks, AR.

you can check which objects are consuming the most memory. There would definitely be a culprit. There have been multiple OOM fixes done since 24.0.2 so it will point us in a specific direction

We are upgrading our UAT cluster to 27.0.0 this week. If there are no issues, I will see if we can upgrade PROD in our next release. Are there any Druid monitors which can detect and log any such memory leak?

Broker memory depends on the queries you are running and the lookups you have, and a lot less on the number of segments (assuming most of them have a similar schema)