Apache Pinot

Screenshot 2022-06-27 at 8.37.42 PM.png

Hi folks. we are using metrics as per <https://docs.pinot.apache.org/configuration-reference/monitoring-metrics> .
Shouldn’t the metric value `max (pinot_broker_queryExecution_95thPercentile{table="$table"})` &gt; `max (pinot_server_totalQueryTime_95thPercentile{table="$table"})` ?

It could be that broker timed out before server could respond.

Screenshot 2022-06-28 at 11.54.16 PM.png

As per my knowledge on broker side, timeout is set in the below order:
1. if query option has timeout, set it to its value.
2. else if table has timeout, set it to its value.
3. else if instance(broker) has timeout, set it to its val.
and based on the above, timeout is put in queryOption map with key `timeoutMs`.

Server will also use the same timeout value.  So both the timeout values are always the same.
I am bit confused how to perceive these metrics. Shouldn’t there be histogram/bucket metric so that we can aggregate and compute percentile metrics?
For example I am comparing max p99 in the below image and still can see broker &lt; server.

Screenshot 2022-06-29 at 12.26.22 PM.png

removed max. I still see broker’s latency is lesser.