Hi All, In some of the MSQ ingestion jobs, the Query Controller is going OOM. Below is the last few lines of the log from the Query Controller:

Query[...] sending out partition boundaries for stage 0 for workers {0, 1, 2, 3}

Query[...] starting 4 workers for stage 1.

Terminating due to java.lang.OutOfMemoryError: Java heap space

There are 5 workers in total for the MSQ job - 1 controller & 4 workers. The heap space allocated is 2GB for each worker along with a direct memory of 500MB. What does the Query Controller keep in memory that would fill up the entire 2GB? Would it help if the number of workers is increased or is the solution only to increase the heap size of each worker? Thanks, AR

You should be able to control this in the

context

"context":{
    "druid.indexer.runner.javaOptsArray": ["-XX:MaxDirectMemorySize=10g", "-Xms2g","-Xmx2g"]
}

Thanks Kyle. I will try setting it in the context and check. Also, can we set -Xms1g -Xmx3g for the peon? Do both of them have to be set to same value?

you would most likely have a big data sketch

Hi Karan, There are no data sketches at all. We are trying to aggregate at daily level. Around 875K records in total. But there are a lot of columns - ~72 columns. And some of them are arrays. Thanks, AR.

There are data sketches used for partition sizes internally by MSQ for partition boundary calculation. Like I said, we would need the heapdump to help further.

We are seeing this issue only in PROD. I will check how to get the heap dump out from there. But is it ok to set "-Xms1g -Xmx3g" for the peon? We want to avoid hitting the OOM suddenly but also don't want to allocate too much memory for the other tasks. Thanks, AR.