This message was deleted.

I think "node 2" is probably referring to a Kafka broker

These disconnections are definitely referring to Kafka brokers. But the brokers themselves show no errors and we’ve deployed an entire new instance of Kafka and the same errors are seen (leading us back to assuming Druid being the root cause somehow).

hmm. i can't think of a reason why druid code would be getting in the way of the kafka client -> broker comms; so, my first thought is let's look into whether there is some problem with the new kafka client would you be able to downgrade the kafka client to 3.1.0 (the version used in 0.23.0) and see if that changes anything?

This is a Kafka client-side (Kafka consumer in the Druid) timeout for Fetch requests (not networking socket timeout). The client will wait for the response to a request for 30 seconds in the current scenario. If the answer is not received before the timeout elapses, the client will resend the request (retry logic). Internally the Kafka client iterates over all the inflight requests and expires any requests that have exceeded the configured request timeout (Kafka consumer logic). The connection to the node (broker) associated with the request will be terminated and will be treated as a disconnection. The issue can be the networking layer, broker utilization, and potentially GC on the Peon JVM. Do you have the client connectivity metrics (like new connections rate) and max request processing time from the broker? Thanks