It is possible that those queries are waiting on the server

Question

It is possible that those queries are waiting on the server side long enough and hit timeout when they start executing

Buchi Reddy · Answer

Exactly my theory That s where I was digging into the code in CombinedPlan where I saw a max no of threads that we can use per query and that depends on the no processors slightly smiling face

Buchi Reddy · Answer

Because we re running in k8s and giving less CPU i m suspecting the threads in the executor could be an issue

Buchi Reddy · Answer

is there a metric or something to confirm that

Mayank · Answer

Search for something like schedularwait

Mayank · Answer

It likely implies you are sending more qps than the provisioned nodes can handle

Buchi Reddy · Answer

I m referring to this constant `org apache pinot core query scheduler resources ResourceLimitPolicy MAX THREAD LIMIT`

Buchi Reddy · Answer

Is that used in default mode

Buchi Reddy · Answer

i m searching for that metric

Mayank · Answer

Number of serving threads is limited to 2x the number of cores iirc

Mayank · Answer

what s the qps you are sending

Buchi Reddy · Answer

it s less than 2 per min it s lt 60

Buchi Reddy · Answer

our CPU allocation was too low Though the server doesn t consumer a lot of CPU some thread pool sizes are initialized capped based on the no of processors available

Mayank · Answer

Yeah in that case with 10k segments I can see how the queue might grow