Hello today I have a problem of refusing service, I think this is a serious problem when the user's request is denied when accessing, should have been required to be pushed into the queue: I tried example in BentoML:

raise ServiceUnavailable("process is overloaded")

bentoml.exceptions.ServiceUnavailable: process is overloaded

I'm setting like this:

@bentoml.service(

name='bentovllm-gemma2-2b-instruct-service',

traffic={'timeout': 300, "concurrency": 32,},

resources={'gpu': 1},

@bentoml.api(

batchable=True, max_batch_size=32, max_latency_ms=200, batch_dim=0

)

Is there any way to set the requests to queue instead of refusing to request?

Is this what you’re looking for? https://docs.bentoml.com/en/latest/get-started/async-task-queues.html

Thank you for support

you’ll need to increase the max_latency_ms to a proper value, 200ms is too small for a batching request