Hi, Is it possible in the cloud to load model from model store dynamically? I want to have single Bento which I can deploy many times with different models (same model code just different versions) I tried to pull inside the service but it fails:

@bentoml.service(
    name="service",
)
class ModelServiceV1:
    def __init__(self):
        model_name = os.getenv("MODEL_NAME")
        model_version = os.getenv("MODEL_VERSION", "latest")
        model_tag = f"{model_name}:{model_version}"

        bentoml.models.pull(model_tag)
        bento_model = bentoml.models.get(model_tag)
        self.model = bentoml.mlflow.load_model(bento_model)

I think adding a BentoCloud api key as env var should work, did you get an authorization error?

yes

Hello, indeed we have a way to access the models. you can do

class ModelService:
    bento_model_ref = bentoml.models.get(BENTOML_MODEL_TAG)
    
    def __init__(self):
         ....

as an example : https://github.com/bentoml/BentoResnet/blob/main/service.py

@Jian Shen Yap thanks, the example you provided is coupling the model to the bento while doing

build

, so it's not possible to have a single bento which you deploy with different models

right. This would be an abstraction choice for BentoML at the moment. Would it make sense if you build 1 bento per model?

we are considering both options right now and we will see what fit us better. since I have 100 models with the same underlying code it's a bit redundant to have the exact same bento per model. for each change need to do many more bento updates

Hi @Matan Perelmuter, pulling models dynamically is a valid option as long as your use case is not super latency-sensitive since the initial model downloading may take a bit of time.

as long as the minimum replicas is 1 it should be reasonable, right?

Just saw your code above - looks like you’re loading the model upon service initialization - it shouldn’t affect the request latency then. I was thinking about dynamically downloading the model at the

api

callback

Would you mind provide us a little more information with your workflow that results you having 100 models? What type of models are they? And is it because of frequent retraining that you have this amount of models? This would provide us a solid case study for future feature development!

we are B2B, so we train a model for each tenant, but the data is normalized and the model code is the same for all of them

Are these classifcal ML models or GenAI model?

tabular data

Cool! understood and thankyou!