This message was deleted.

Not sure that this is what you are running into : https://github.com/apache/druid/issues/4018 but in that reported issue the problem seemed to originate from some other (non-Druid) service using the same port as the historical (8083) (it was Debezium in that case). So that when the broker tried to communicate with the historical it was actually talking to Debezium. Are you running anything else? Could you be having a port conflict?

We checked that Sergio and no other services are running other then druid on 8083

Hi Satish. Are you seeing bottlenecking on the brokers?

We see all the resources available On brokers . We had issues with ram and cpus we added more brokers and we have 4 brokers behind LB and all looks healthy

Thanks for that. Queries are the other thing that come to mind. Have any queries changed recently? Flame graphs might be worth a look.

You mentioned it happens when "running concurrent queries". Could you elaborate? Does it happen as soon as you run two concurrent queries? or what is the pattern?

We running around 4000 concurrent queries, we r seeing this after 3000 plus queries

Okay. I think a review of threads and connection parameters may help. Some relevant parameters and rules from the basic cluster tuning guide:

For Historicals, druid.server.http.numThreads should be set to a value slightly higher than the sum of druid.broker.http.numConnections across all the Brokers in the cluster.

On the Brokers:

druid.server.http.numThreads on the Broker should be set to a value slightly higher than druid.broker.http.numConnections on the same Broker.

If you are querying from real-time ingestion on Middle Manager Tasks:

For Tasks, druid.server.http.numThreads should be set to a value slightly higher than the sum of druid.broker.http.numConnections across all the Brokers in the cluster.

If none of these are an issue perhaps you are having trouble with some other resource contention on CPU or Memory. How is the utilization level on the Brokers, Historicals and MMs when this situation occurs?