Hi, our brokers are taking 25s (max time) to return queries, but direct server queries return instantly. I took heap dumps, pmaps, etc. and the one thing that stands out is jstack output. Looks like HelixTaskExecutor threads are all waiting on the same object. Anyone ever see this behavior?

Helixtaskexecutors are not in query path

Seemed fine, anything in particular?

Can you find the queries with high latency from the logs?

I see that not all the servers responded, ex. 2/3 servers responded in 25s

So the problem is on server side

We have recently seen helix threads waiting for queue full in the server in Linked. These things lined up with other problems, so we were not sure if this was a helix issue or a pinot issue. I tend to believe that we have a pinot issue when trying to download segemnts from the controller, but I could be wrong. @Sidd investigated this most recently, but he is away on a holiday and may not respond until next year. We will watch out for more at our end. Meanwhile, is this reproducable? Is this offline only or offline+realtime or realtime only use case

It was for all tables on the server. Seems to happen after the server is running for about 1 week or so. I don't think it was the helix threads, after restarting they all appear to be in

WAITING

state. All I see is that 1 server can do that to the entire cluster but when you submit an AsyncServerRequest via presto the servers respond instantly.