I did some reading on mmap and it does look like the metrics are accurate.In our case we have 4GB of RAM available for processing 7GB of segments.Incase all segments are needed then this will lead to latency .

POST  <http://controller-pinot.dev.dp.ctct.io/tables/recommender>

gives the option of adjusting the Query Latency. I was also going through Ankit Sultana's Pinot Table joins talk and he mentioned that more than query latency it is the number of concurrent queries executing will give us a better estimate