has anyone ever encountered random 503s in your pinot setup? we get them every once in a while just trying to understand and debug what may be happening since the dashboards are not telling me anything out of the extraordinary, we get this 503s sometimes when we hit the broker, all this setup is kubernetes so we have a loadbalancer on top of it, and have 2 brokers running, any ideas?

Any logs in broker/server for the request?

subscribing because i’ve seen this as well, but also haven’t dug into the logs since it’s 10s of 10ks requests

hard to find them since it’s so low @Mayank i don

There’s a metric for ’schedularWaitTime` (time for which query was sitting queue). Check if that is spiking

i get the error on the client so i have those logs i know how the request looked like, as in the sql, but hard to correlate it to the logs

before logs, you can check for metric above, along with cpu/mem/latency metrics

that’s a server metric?

Yes ms. Is this p99? If So seems like this this is fine, may be broker needs to be scaled

that’s the mean rate p99 is just 1

All this seems healthy to me. Could there be a load balancer issue?

update: I do see 503s in the load balancer but what could be the reason the load balancer may be 503ing?

What’s the overall qps and total number of brokers?

2 brokers 100qps

Hmm, that doesn’t seem too much for 2 brokers. Not sure.