Hi Druids! Some of our compaction and index parallel ingestions are failing, post a version upgrade from 27 to 30. We're seeing a recurring log of unable to delete temp Could not clean temporary segment file: /opt/druid/var/druid/task/slot6/index_parallel_chart_982-v33-accuracy-dn-model_type_galfdnoo_2024-09-23T052006.136Z/work/indexing-tmp/chart_982-v33/2023-02-01T000000.000Z_2023-03-01T000000.000Z/2024-09-23T052048.727Z/0, When looking into the task slot 6 folder inside the MM pod I am not seeing any files, so it looks like some process is trying to delete a file that does not exist. Has anyone faced this issue? Any help will be appreciated

see same problems in log. I can't say what is the reason or how to solve it

that warning itself shouldn't cause a task failure. something else must be going on too

Have increased heap size for the main MM process and for independent indexer processes. Post that seems to be working, will keep the thread updated.

Can you please check if your tmp directory has enough space , if task is failing with

No space left on device"

it might be due to space issue while intermediate disk persist. MM runtime and JVM configs.

druid.indexer.task.baseTaskDir=var/druid/task

-<http://Djava.io|Djava.io>.tmpdir=var/tmp

Hi yes I am seeing that temp dir is currently at 100%, we're gonna be changing the temp dir size. But should temp dir be able to fill 150GB directory? Is something going wrong for that to be able to happen?

Is the temp dir being used for anything else apart from the Druid process? Can you please check the top culprits filling up the temp directory?

Hey Laksh, I am seeing a large number of java_pid1481.hprof type files. These seem to be java heap crash files that are taking upto 70GB. I think these should be safe to delete but is there anyway to make crash files not print or to not dump here.

which java process is generating the hprof files? You can modify the JVM arguments to not generate the heap dump on exit.

I am not sure how to analyse this Hprof file, the file was generated in the MM processes' configured temp dir

To identify which process generated that file: If the process is no longer running, you can check the system logs to see if any Java processes were running around the time the heap dump file was generated.

Druid MM process args druid_indexer_runner_javaOptsArray: '["-server", "-Xms2400m", "-Xmx2400m", "-XX:+IgnoreUnrecognizedVMOptions", "-XX:+UseG1GC","-XX:+PrintGC","-XX:+PrintGCDateStamps","-XX:+ExitOnOutOfMemoryError","-XX:+HeapDumpOnOutOfMemoryError","-XX:HeapDumpPath=/opt/druid/var/druid", "-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-Djava.io.tmpdir=/opt/druid/var/druid","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager","--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED","--add-exports=java.base/jdk.internal.perf=ALL-UNNAMED","--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED","--add-opens=java.base/java.lang=ALL-UNNAMED","--add-opens=java.base/java.io=ALL-UNNAMED","--add-opens=java.base/java.nio=ALL-UNNAMED","--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED","--add-opens=java.base/sun.nio.ch=ALL-UNNAMED","--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED", "-XX:MaxDirectMemorySize=1500m"]' This is the exact path where the heap dumps were happening, will remove the HeapDumpOnOutOfMemoryError and HeapDumpPath to prevent this issue, is there any alternate option folks use for OOM errors logging?

either remove XX:+HeapDumpOnOutOfMemoryError if you don’t want it to generate the heap dump, or change -XX:HeapDumpPath=/opt/druid/var/druid to another location where there isn’t a lack of availability of the free memory

Okay, I just reported this same issue so I have more details. When the job completes (successfully or failure) then the files are cleared. But until the jobs completes it will leave these temporary files around and spew warnings about being unable to clean them. https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1727447450295289

Hey @Laksh Singla when we removed the option for XX:+HeapDumpOnOutOfMemoryErro and -XX:HeapDumpPath=/opt/druid/var/druid, we started seeing the issue with the following error : java.lang.RuntimeException: org.apache.druid.msq.indexing.error.MSQException: WorkerFailed: Worker task failed: [query-d1312a9a-f2bd-407d-8a7a-00a560a74e80-worker13_0] (Failed to get storage slot due to error [Unable to pick a free slot, this should never happen, slot ...) at org.apache.druid.common.guava.FutureUtils.getUnchecked(FutureUtils.java:83) at org.apache.druid.msq.indexing.MSQWorkerTaskLauncher.launchTasksIfNeeded(MSQWorkerTaskLauncher.java:253) at org.apache.druid.msq.exec.ControllerImpl$RunQueryUntilDone.startStages(ControllerImpl.java:2979) at org.apache.druid.msq.exec.ControllerImpl$RunQueryUntilDone.run(ControllerImpl.java:2724) at org.apache.druid.msq.exec.ControllerImpl$RunQueryUntilDone.access$000(ControllerImpl.java:2681) at org.apache.druid.msq.exec.ControllerImpl.runTask(ControllerImpl.java:433) at org.apache.druid.msq.exec.ControllerImpl.run(ControllerImpl.java:372) at org.apache.druid.msq.indexing.MSQControllerTask.runTask(MSQControllerTask.java:258) at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179) at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478) at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.druid.msq.indexing.error.MSQException: WorkerFailed: Worker task failed: [query-d1312a9a-f2bd-407d-8a7a-00a560a74e80-worker13_0] (Failed to get storage slot due to error [Unable to pick a free slot, this should never happen, slot ...) at org.apache.druid.msq.exec.ControllerImpl.lambda$initializeQueryDefAndState$2(ControllerImpl.java:739) at org.apache.druid.msq.indexing.MSQWorkerTaskLauncher.checkForErroneousTasks(MSQWorkerTaskLauncher.java:603) at org.apache.druid.msq.indexing.MSQWorkerTaskLauncher.mainLoop(MSQWorkerTaskLauncher.java:391) at org.apache.druid.msq.indexing.MSQWorkerTaskLauncher.lambda$start$0(MSQWorkerTaskLauncher.java:172) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ... 3 more We have a MM autoscaler that looks at pending tasks and then increases the MM count, but the autoscaler was not able to get the pending tasks from the overlord. On re adding the above 2 options, the scaling started working correctly. Is there any light you can shed on this issue?