This message was deleted.

Full stack trace for reference:

2023-10-19T22:37:23,941 ERROR [processing-1] org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2 - Exception with one of the sequences!
java.lang.RuntimeException: QueryInterruptedException{msg=null, code=Query interrupted, class=java.lang.InterruptedException, host=null}
	at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:120) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:102) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_372]
	at org.apache.druid.java.util.common.concurrent.DirectExecutorService.execute(DirectExecutorService.java:81) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_372]
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58) ~[guava-16.0.1.jar:?]
	at org.apache.druid.query.DirectQueryProcessingPool.submitRunnerTask(DirectQueryProcessingPool.java:42) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.ChainedExecutionQueryRunner$1.lambda$make$0(ChainedExecutionQueryRunner.java:100) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at com.google.common.collect.Iterators$8.transform(Iterators.java:794) ~[guava-16.0.1.jar:?]
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-16.0.1.jar:?]
	at com.google.common.collect.Iterators.addAll(Iterators.java:357) ~[guava-16.0.1.jar:?]
	at com.google.common.collect.Lists.newArrayList(Lists.java:147) ~[guava-16.0.1.jar:?]
	at com.google.common.collect.Lists.newArrayList(Lists.java:129) ~[guava-16.0.1.jar:?]
	at org.apache.druid.query.ChainedExecutionQueryRunner$1.make(ChainedExecutionQueryRunner.java:92) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:98) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:185) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:165) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:252) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:239) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_372]
	at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:251) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]
Caused by: org.apache.druid.query.QueryInterruptedException
	at org.apache.druid.query.BaseQuery.checkInterrupted(BaseQuery.java:53) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter$IncrementalIndexCursor.reset(IncrementalIndexStorageAdapter.java:423) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter$IncrementalIndexCursor.<init>(IncrementalIndexStorageAdapter.java:331) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter.lambda$makeCursors$0(IncrementalIndexStorageAdapter.java:280) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:42) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.Sequence.toList(Sequence.java:87) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:112) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	... 38 more
Caused by: java.lang.InterruptedException
	at org.apache.druid.query.BaseQuery.checkInterrupted(BaseQuery.java:53) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter$IncrementalIndexCursor.reset(IncrementalIndexStorageAdapter.java:423) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter$IncrementalIndexCursor.<init>(IncrementalIndexStorageAdapter.java:331) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter.lambda$makeCursors$0(IncrementalIndexStorageAdapter.java:280) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:42) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.java.util.common.guava.Sequence.toList(Sequence.java:87) ~[druid-core-25.0.0-sfmc.jar:25.0.0-sfmc]
	at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:112) ~[druid-processing-25.0.0-sfmc.jar:25.0.0-sfmc]
	... 38 more

What version of druid are you using? This might or might not be helpful, but I think there were multiple causes, and also some fixes in newer versions. I think I saw it come up more at some point (I forget which version), then less after some fixes.

Thanks for the response @Ben Krug were on 25.0.0

@Andrew Ho do you see any nodes restarting at the same time? I looked for other reports, and in some cases, threads got overloaded. Is

druid.query.scheduler.numThreads

set to less than

druid.server.http.numThreads

on your different nodes? If not, the scheduler can take up all threads and health checks can't get one, so the node can get cycled (if you have any health checks enabled).

@Ben Krug No, we are not seeing any node restarts. We are using the default value for

druid.query.scheduler.numThreads

(Unbounded). When I look at the task logs I see the above query failure exception. But after that the ingestion task continues processing fine

OK, sorry I can't say for sure what's going on. Good that the ingestion works, and hopefully query retries succeed. If it gets worse, then possibly looking at setting druid.query.scheduler.numThreads (to, eg, druid.server.http.numThreads-1) might be worth trying.

Thanks @Ben Krug Are there any parameters we can tune so that the Broker retries these automatically?

I'm not sure... For historical data, if the broker fails to hear from one historical, I think it might try others for the segments. I don't know, but it's possible that if you have multiple replicas for realtime, it might do the same. As far as other retries, I think they might have to be done client-side in any code. But you might ask that as a new question/thread, and see whether others have experience with how to automate retries.

Thanks @Ben Krug Would you recommend cross posting this in #C0309C9L90D?