Hello everyone, in Bentoml there is a batch processing mechanism to handle many requests simultaneously. For example, if I set a batch size 32, when there are 50 requests sent to Bentoml, there is a Scale Instance automatic mechanism to allocate GPU resources for it. I do not use Bento Cloud, only use K8S and packaged containers

Sorry i did not understand your question.

there is a Scale Instance automatic mechanism to allocate GPU resources for it.

what does this means?

My question is: Does Bento support automatically scaling instances when the batch size reaches the request limit?

for open source BentoML, the unit of distribution is a docker container, so you will have to manage your own scaling orchestration. I believe it is the same as FastAPI. Our batching mechanism allows you to batch multiple request to fully utilize your GPU, which otherwise you will need to implement it yourself in FastAPI

the batching is not related to autoscaling, requests exceeding the max batch size will be queued up

Thank you for support, i understood it

Hi can any1 help me to understand this

Hi CA PRITI PATEL, sorry for response lately, this problem is not autoscaling in batch processing mode, in fact, it not related to, we have to set up it ourselves when deploying it on K8s