hey guys, just wanna ask, can it be, that customized endpoint with path parameter breaks the

/metrics

? so I have a customized FastAPI endpoint in Bentoml Service

GET /embedding/{id}

• in my test environment because the

id

s are very few, so I can see the default metrics from

/metrics

• but in my production environment the

id

s are millions, and I can’t get metrics anymore from the

/metrics

endpoint… • I see that the metric are labeled with endpoint, so my guess is that if the URL has too many path parameters, the cardinality of the metric is becoming too large to be queried, is this the root cause? • any suggestions to fix this?

when you say you cannot query them, what do you mean? isit a 4xx/5xx error? or the query from swaggerUI is not responsive? swaggerUI is not very good with handling large outputs

i got

Error: socket hang up

if use postman to send the request and i don’t get anything in my prometheus or grafana, says no data

i think its possible that the return outputs are too huge and exceeds the http request limit. Do you think you can verify this?

not sure how.. but it’s a good hint.. the result can indeed be really large

Is there a way for you to limit the number of endpoints?

well, yes, but then i need to change endpoint contract

Maybe you wnat to simulate locally and see if it breaks when your endpoint reach a certain amount

i can also do that