Hi All, In our Druid cluster, we are seeing some odd exceptions when submitting the MSQ jobs through the API

/druid/v2/sql/task

. Sometimes we see a "504 Gateway Timeout" exception. Other times we see a "Task [] already exists" exception. We can see the TimeoutException in the router logs as well. But we are unable to see any issue in any of the other services that would give a pointer to why this is happening. Can someone suggest what could be the issue? Druid version: 27.0.0 Thanks, AR.

if there's a timeout on that API, it might be that the Broker or the Overlord is at capacity. It should ordinarily return more-or-less immediately

Thanks Gian. We have configured the Co-ordinator to function as Overlord as well. We don't see any exception in the Coordinator logs. Also, I couldn't find any explicit thread pool config for the Co-ordinator/Overlord. Does communication b/w the router/broker & Overlord happen over HTTP or ZK? Regards, AR.

the router/broker -> OL communication is largely over HTTP. the only thing that happens over ZK is service discovery

Thanks Gian. How can we check if Broker or Overlord is running out of capacity? Also if it were out of capacity, why would the request time out (504 Gateway Timeout) at the Router? I would assume it would throw something like a "Request rejected" exception if there were no free threads to handle a request. Thanks, AR.

typically, they will queue requests rather than reject them if they're at capacity. which means typically being at capacity leads to timeouts