This message was deleted.

This implies the default bento wouldn’t be customized at all: • no custom preprocessing • No custom post processing • No instrumentation: auth, custom headers, custom metrics/logging etc.

It's definitely something we've considered! Certainly sounds appealing when you put it like this, I think there would be a decent amount of value on autogenerated / default bentos. I'll bring this up!

Actually I think it would be more interesting if this could be done by generating the

service.py

file using a jinja template, and potentially allow users to customize the jinja template (at least the blocks) to allow devops engineers on the team to inject useful things e.g. feature stores, custom readiness probes etc. My team built an internal package that generates a

service.py

file, then builds a bento, based on a predict function (and various other inputs like pydantic models), but some things are very awkward to handle • How does the DS specify imports used in the predict function - currently we just pass a list of strings and then the imports will be injected through the jinja template • How does the DS specify additional functions they need to use in their predict function - currently, we have no way to do this, you just write the whole logic in the predict function We built this primarily because it is very awkward to try to create a bento from a Databricks environment, where creating the bento folder structure is non-trivial as the filesystem that a DS typically sees exists in the Databricks control plane and not the actual compute environment where their notebooks run. We also wanted to inject a lot of custom functionality into all our models, and we didn't want the DS to have to worry about all these.

@jiewpeng nice! I kind of see our two comments as different asks: 1. a totally vanilla "default bento" 2. a templatized "default bento" that is overridable by the DS or their ops team I still think there's value in [1], but certainly agree that I'll end up needing to do what you described [2] for other use cases. Probably, we'll end up doing the same thing as you--making our own parameterized template to add our own opinions the bentos that the DS build.

If you run

bentoml build

or

bentoml.bentos.build()

in python, you get a bento which you can choose to deploy via whatever means, so that already achieves what we want. So long as we can build a "default bento" i.e. generate the files required for

bentoml build

and then package the bento, I think that already has some value. Personally I think [2] has a lot more value in actual production use, because not being able to override anything typically means that you need to engineer a lot of things around the constraints of the default bento. I think [1] has some value in very generic use cases like deploying a vanilla LLM / model, but most use cases would need [2] in order to define basic preprocessing and postprocessing steps. [2] is also required to run models reliably in production: for example, without customising the readiness probe, you will get requests routed to pods that are not ready to accept traffic when you eventually decide to update the model and redeploy it.

If you run

bentoml build

or

bentoml.bentos.build()

in python, you get a bento

Doesn't

bentoml build

require a

bentofile.yaml

and

service.py

though? I was thinking this would avoid having to write those.

I think [1] has some value in very generic use cases like deploying a vanilla LLM / model, but most use cases would need [2] in order to define basic preprocessing and postprocessing steps

Agreed. [1] basically puts bentoml at feature parity with MLEM. Not great for every scenario, but has some use. Possibly related to [2], it'd be kind of nice if there were a

bentoml init

command that walked you through a wizard and generated the project boilerplate needed to get started with a bento project. I find myself reviewing projects and copy/pasting them together each time I want to get started. Not an issue for a team that has their own CookieCutter set up, but I think the lack of such a feature adds friction to newcomers trying to get onboarded to bentoml.

Doesn't

bentoml build

require a

bentofile.yaml

and

service.py

though? I was thinking this would avoid having to write those.

bentoml build

does, but

bentoml.bentos.build()

builds the

bentofile.yaml

for you based on the arguments passed into it. You still need the

service.py

file, which what we have in place is something that writes all the boilerplate, then saves the

service.py

file in a temporary directory, then calls

bentoml.bentos.build()

, specifying the temporary directory as the build context.

Agreed. [1] basically puts bentoml at feature parity with MLEM. Not great for every scenario, but has some use.

Agree with this and it can be a first step, but perhaps we could think of implementing it in a way that extends to being able to do [2]? We chose bentoml because it feels the closest to being built for a DS (and not devops engineers) to use, and also allow sufficient flexibility for both DS and ops. However the interface is still awkward because the DS has to build a separate

service.py

and

bentofile.yaml

file. If we can solve the issue of generating these 2 files, and also make it work in a notebook-based development environment, I think it would be great. Currently we build our own system on top of bentoml to achieve this, but having it centrally maintained within bentoml would be ideal.