BentoML

Our model is 2GB in size, so we’re concerned it’s going to take a while to pull the docker image containing the model when we go to deploy

yes it is possible!  the default workflow with built-in runners will always package the model together with the container. However with custom runner, user can implement their own model loading logic <https://docs.bentoml.com/en/latest/concepts/runner.html#custom-runner>

in this case, you can define where to load the model, and dynamically mount the model directory during deployment

Hmm, okay I see. Does this approach also make sense:

• use `bentoml model export` to export a model itself and upload to s3
• In our deployment, have a script to download the model from s3, and in the Dockerfile:
• `bentoml model import` the downloaded model
• `bentoml build` to build the project, which references the model we just imported
• `bentoml serve` to serve the model
This would avoid having to write our own custom runner, but comes at the cost of having to build the model on the startup of the deployment which also isn’t ideal. What do you think?

Wanted to bump this just to verify approach makes sense! ^