Hello everybody, Could somebody point me to the root-cause/fix of task starting failure of batch ingestion job that happens very rarely? Still it's annoying because is a back-filling task for historical datasources. You can find attached the full task logs but this is the error stack trace:

2026-04-14T20:05:06,869 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up.  Failing.
java.lang.reflect.InvocationTargetException: null
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:137) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:390) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.Main.main(Main.java:112) [druid-services-34.0.0.jar:34.0.0]
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at org.apache.commons.io.function.Uncheck.wrap(Uncheck.java:339) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:199) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more
Caused by: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148) ~[?:?]
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:226) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323) ~[?:?]
	at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71) ~[?:?]
	at java.base/java.nio.file.Files.walk(Files.java:3918) ~[?:?]
	at org.apache.commons.io.file.PathUtils.walk(PathUtils.java:1847) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.streamFiles(FileUtils.java:2964) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.lambda$listFiles$10(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:197) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more

A task that re-ingest data from other one datasource (realtime) to another (historical-hour) cannot start because it doesn't find some files of a compaction task shuffled data that is associated with a completely unrelated datasource. The task doesn't even overlap in time, the compaction task finished around 9 minutes before the failed re-indexing task even started. Thank you!