This message was deleted.

Seeing lots of

java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms
	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
	at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_275]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]

But it seems expected for jetty's behavior?

What about the broker's

druid.broker.http.numConnections

? From the docs:

On the Brokers, please ensure that the sum of druid.broker.http.numConnections across all the Brokers is slightly lower than the value of druid.server.http.numThreads on your Historicals and Tasks.

druid.server.http.numThreads on the Broker should be set to a value slightly higher than druid.broker.http.numConnections on the same Broker.

Could your query workload on the tasks be saturating its http threads?

On this cluster, we have 15 numThreads for each broker and there are two brokers at the same time for now.

Is zk a bottleneck? I've seen something similar, not quite the same error, where tasks were toiling away but didn't return their status so replacements start, extra tasks cause more load, sadness ensues. We ended up seeing a dramatic improvement switching from zk to httpRemote on the overlord (might become the default soon?) https://github.com/apache/druid/pull/13092

@Peter Soderberg It does not seem to me ZK is the bottleneck. I don't see zk errors or warns in zk logs. We do have running tasks that return success. If zk has issues, I would expect our ingestion to be impaired, but it does not seem the case.