This message was deleted.

It seems you are using hash partitioning. I would advise you to use to range partitioning as it gives a better distribution of data amongst segments and also prevents such OOM failures.

I tried but had no luck with the

maxNumSegmentsToMerge

, I’m currently running with range partitioning to see it it handles the indexes better

A couple comments: 1) I'm curious where exactly the task is running OOM. If you manage to capture a heap dump, the heaviest objects in the heap dump generally tell us something about what to do (to do it: use

-XX:+HeapDumpOnOutOfMemoryError

in your

druid.indexer.runner.javaOpts

or

druid.indexer.runner.javaOptsArray

) 2) As an alternative approach, in the last release (24.0) we introduced SQL-based ingest and multi-stage query tasks which is totally re-engineered and much better at memory management. This can be used for manual compaction using REPLACE: https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#overwrite-data-with-replace. There are some examples: https://druid.apache.org/docs/latest/multi-stage-query/examples.html. If you use it, would love your feedback on this new feature! Note: we haven't yet built a way to use this for autocompaction, but we plan to in the future.

uhh, I’m really interested in that second one. I just set up MSQE for a different part of the project so it is relatively lightweight to test

great to hear!