Hey folks!
We're facing an issue during compaction...
# generalu
Utkarsh Chaturvedi
10/03/2024, 6:24 AMHey folks!
We're facing an issue during compaction where we run out of heap memory. The pattern being seen everytime is that the in memory data gets flushed to disk and then the GC starts which breaks in the middle due to OOM issue. Attaching a log for the same. This has been happening fairly consistently since we've upgraded to v30. Has anybody else faced such an issue? Is there anything to be done besides increase heap memory for the peon process?
Utkarsh Chaturvedi
10/03/2024, 9:09 AMIn this specific thread I am seeing that the index_parallel ingestion triggered by the compaction task is what causes the Out of heap memory issue. Druid cluster tuning states that tasks should have heap memory of about 1GB, we've given 2.4 GB. Still we're facing this issue. The overall size of the data source is not extremely high, so we shouldn't really be facing this issue IMO