This message was deleted.

The overlord coordinates directly with each task using REST calls, it seems like it is unable to communicate with it. How is this deployed? Could it be a network issue? Can you share the task log for task "index_kafka_sn-content-events_60625f27846ee46_aehhhela"?

Hi @Sergio Ferragut I thought it was some kind of network issue too. I can see that the task starts up listening on port 8100 but in the coordinator log I see that connection refused message. In order to test network connectivity I started a simple java process that listens on 8100 on the vm where the peons run and was able to telnet from the coordinator machine ok. Heres the log you asked for... if you look for the setting 8100...

seanlaval@Seans-MBP Downloads % grep 8100 log
2023-01-06T16:24:34,847 DEBUG [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='druid/middleManager', host='w3-druid-data-01-preprod', bindOnHost=false, port=-1, plaintextPort=8100, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}]
2023-01-06T16:25:23,155 INFO [main] org.apache.druid.cli.CliPeon - * druid.plaintextPort: 8100
2023-01-06T16:26:41,489 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8100]
2023-01-06T16:26:44,893 DEBUG [main] org.eclipse.jetty.util.component.ContainerLifeCycle - ServerConnector@221cdd87{HTTP/1.1, (http/1.1)}{0.0.0.0:8100} added {JettyMonitoringConnectionFactory@5d202dc4{STOPPED},AUTO}
2023-01-06T16:26:44,893 DEBUG [main] org.eclipse.jetty.server.AbstractConnector - ServerConnector@221cdd87{HTTP/1.1, (http/1.1)}{0.0.0.0:8100} added JettyMonitoringConnectionFactory@5d202dc4{STOPPED}
2023-01-06T16:26:44,893 DEBUG [main] org.eclipse.jetty.util.component.ContainerLifeCycle - Server@451a4187{STOPPED}[9.4.48.v20220622] added {ServerConnector@221cdd87{HTTP/1.1, (http/1.1)}{0.0.0.0:8100},AUTO}
2023-01-06T16:26:59,930 DEBUG [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void org.apache.druid.java.util.emitter.service.ServiceEmitter.start()] on object[ServiceEmitter{serviceDimensions={service=druid/middleManager, host=w3-druid-data-01-preprod:8100, version=24.0.1}, emitter=NoopEmitter{}}].
2023-01-06T16:27:09,450 DEBUG [task-runner-0-priority-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_sn-content-events_60625f27846ee46_aehhhela] location changed to [TaskLocation{host='w3-druid-data-01-preprod', port=8100, tlsPort=-1}].

Another possibility is the http threads. Do you have queries running on the real-time data? The brokers also communicate with the tasks to resolve queries and use up http threads to do so. Your can increase it with druid.indexer.fork.property.druid.server.http.numThreads