This message was deleted.

Do you want to swap in and out new models on the fly without rebuilding? I think this story is covered by Bento CRM in Bento Cloud; @Xipeng Guan is that right?

Essentially I am just trying to see if a deployed Bento requires to have models baked into it, or whether it can ingest new models and serve them. By the way, the implementation I posted seems to work completely fine. At least when running locally in a container. I was just wondering if this method makes sense and whether there is a standard way of achieving this.

Right, yeah, I think that the canonical way is built into Bento Cloud. What you've done is fine, I think, it just means that the model will be running in the same process as your API service worker, which may or may not be desirable. To do it fully properly, you'd have to have management outside of the BentoML API.

Hmm I see. What if I make a custom

Runner

which is able to ingest a new model. So for example, I would add model to the service in the same way, but then rather than creating a new runner to handle that model, I just add it to an existing runner. Would this enable the model to be run in the Runner worker rather than the API?

Yes, I think that would work!

Aha, thanks! I will give that a go. I guess an advantage of that is that I can configure that runner in a standard way as well (using the

yaml

as per usual at launch time of the bento). Thank you for your answers, this has been very helpful! 🙏

@sauyon @Xipeng Guan You're mentioning 'swapping in and out models on the fly without rebuilding' -> this is a problem I'm trying to solve. You're also mentioning this is solved in Bento Cloud.. could you point me towards the right resources? Thanks in advance 🙂

Sadly it seems I was mistaken about this, although the feature is coming it's not currently available. Sorry about that!