It is possible that those queries are waiting on t...
# troubleshootingm
Mayank
07/22/2020, 5:15 AM
It is possible that those queries are waiting on the server side long enough, and hit timeout when they start executing
b
Buchi Reddy
07/22/2020, 5:18 AMExactly my theory. That’s where I was digging into the code in CombinedPlan, where I saw a max no. of threads that we can use per query and that depends on the no. processors 🙂
Buchi Reddy
07/22/2020, 5:18 AMBecause we’re running in k8s and giving less CPU, i’m suspecting the threads in the executor could be an issue
Buchi Reddy
07/22/2020, 5:18 AMis there a metric or something to confirm that?
m
Mayank
07/22/2020, 5:18 AM
Search for something like schedularwait
Mayank
07/22/2020, 5:19 AM
It likely implies you are sending more qps than the provisioned nodes can handle
b
Buchi Reddy
07/22/2020, 5:20 AMI’m referring to this constant.
org.apache.pinot.core.query.scheduler.resources.ResourceLimitPolicy#MAX_THREAD_LIMITBuchi Reddy
07/22/2020, 5:20 AMIs that used in default mode?
Buchi Reddy
07/22/2020, 5:20 AMi’m searching for that metric
m
Mayank
07/22/2020, 5:21 AM
Number of serving threads is limited to 2x the number of cores iirc
Mayank
07/22/2020, 5:21 AM
what's the qps you are sending?
b
Buchi Reddy
07/22/2020, 5:23 AMit’s less than 2. per min it’s < 60
Buchi Reddy
07/22/2020, 7:12 AMour CPU allocation was too low. Though the server doesn’t consumer a lot of CPU, some thread pool sizes are initialized/capped based on the no. of processors available.
m
Mayank
07/22/2020, 12:26 PM
Yeah, in that case, with 10k segments, I can see how the queue might grow.