Hi, I have a question about metrics and Grafana. We are using BentoML 1.2.7 and are trying to monitor the average number of requests over various periods, such as one hour, 5 minutes, or a few days. We've tried the following PromQL queries:

promql

avg(bentoml_service_request_duration_seconds_count{namespace="$namespace", job=~"$job", exported_endpoint!="/", exported_endpoint!="/docs.json", exported_endpoint!="/robots.txt", exported_endpoint!~"/static_content/.*"}) by (runner_name)

and

promql

avg(bentoml_service_request_duration_seconds_sum{namespace="$namespace", job=~"$job", exported_endpoint!="/", exported_endpoint!="/docs.json", exported_endpoint!="/robots.txt", exported_endpoint!~"/static_content/.*"}) by (runner_name)

However, the results seem off We are seeing averages of 3-5 minutes, whereas our logs show that the requests take less than 1 minute. Any idea what we might be missing or doing wrong? Thx in advance

It may be hard to see what is happening here. Do you have a simple reproducer for your this scenario that we can look at?

Hi @Jian Shen Yap, thanks for the quick response. I've pasted the PromQL we've tried to use. We need a simple PromQL query for the average count of requests to BentoML in our Grafana dashboard.

have you looked at whether the individual metrics make sense, i.e.

bentoml_service_request_duration_seconds_sum

from the

/metrics

endpoint? i would suggest you to fire a simple request and see whether the results agree with observation

Yes, we have attempted using bentoml_service_request_duration_seconds_sum and have also tried dividing bentoml_service_request_duration_seconds_sum by bentoml_service_request_duration_seconds_count.

Hi, i am not able to reproduce this. Can you help with sharing the exact steps? 1. The service file that you use 2. Your BentoML version 3. The results from

/metrics

and prometheus that you see. Expected vs actual.

@Jian Shen Yap: The issue was resolved using bentoml_service_request_duration_seconds_sum expression / bentoml_service_request_duration_seconds_count.