Hi! Serving BentoML on an AWS instance (with multiple models in a runner, CPU VM) • We have asynchronous functions, when sending multiple requests it seems to stop (probably because it cant serve 10 requests at a time) • Is there a way to queue things when running as

bentoml serve

or limit requests?

What’s the target concurrency of your API?

probably 2 or 3 requests being handled

there’s a request queue backlog already, are you looking for async endpoint - submitting a task, get a task id, and using task id to query result?

sorry i just came back for this

see the links above https://docs.bentoml.com/en/latest/bentocloud/how-tos/autoscaling.html#external-queue

Hi, when using non-Bentocloud BentoML server instance and the concurrency is 1 and max_concurrency is also 1 with a timeout of 60s. What happens if there are two requests simultaneously? Does it get queued and if 60s is not finished for the first request, the 2nd request receives a timeout?

it will reject the request with “429 too many requests”, which is expected behavior

thanks @Chaoyu what would be the ideal settings for just allowing one request at a time for an async function? the queue mechanism should be at the backend calling it, right? another question, can this setting

traffic={
         "concurrency": 32,  # An integer value
        ...
    }

be done in inside here?

service = bentoml.Service(
        "inference_graph",
        runners=model_runnable.service_runners,
    )