This message was deleted.

Hey! Separating model from the image is something we are working to make it idiomatic to BentoML. The current way that we suggest is to load model on startup is loading the model in the constructor of the Runner. This will require you to write a custom Runnable

Thank you, this is exactly what I did. would I be able to inherit somehow from the diffusers runner and extend it?

By default bentoml serve run in production mode. In this case we spawn # of api_server = ur cpus core. Development mode only spawn 1 api server with pdb support, so you can throw a breakpoint in there for debugging

Hi @Jian Shen Yap, does separate model loading on start has a feature request, is it planned to be released soon? Do you have any examples for mixing custom runner with existing one? Thanks!

would I be able to inherit somehow from the diffusers runner and extend it?

Theoretically this could be possible, that would require you to overwrite the constructor of the runner to load model from somewhere outside of the docker image. we currently don't have any examples of that, but please give it a try and we are happy to answer some of the questions that might not be apparent to you.

This feature does not yet have an ETA. We are still collecting more requirements. At the current state, a custom runner should be suffice for pulling and loading the model at the runner’s init. @Amit Gelber do you see any gaps of this approach? Your feedback will help us prioritize this work.

Hi @Sean! no, no gaps, at the moment this is the approach we implemented. the only downside is "losing" the diffusers integration. I would need to investigate how to mix both.

That’s valuable feedback. We will look into how to preserve the framework integration at the same time allowing to load the model at initialization.

thanks.