BentoML

Hello! Wondering if there might be any interest in adding this example to the gallery?
<https://github.com/yuhongsun96/gallery/blob/master/pytorch/pretrained/ServiceDemoSBERT.ipynb>

I saw some comments from other people wondering if this is possible or having trouble doing something similar (pretrained models).

 the upcoming new version will support integrating with any asgi python web server, including fastapi 

Hi <@U02LA7EFM0T> - would you be interested in helping to contribute this example with the new 1.0 branch’s API? We are bringing a new API specifically for importing pre-trained models from model hub

Sure thing, would love to help!
Couple questions,
Are the implementations/APIs heavily reworked for 1.0 or is it mostly what I've been seeing in current gallery repo?
What I did above was from sentence_transformers (built on top of transformers) as provided by huggingface, it's not model hub, is that ok?
If you want me to demo an example from model hub I can also do that. If you guys want a sample for every DL library like pytorch, tensorflow etc, it may take a while to go through them all though.
What specific features/functionalities do you want covered?

Looking forward to contributing :)

<@U02LA7EFM0T>
&gt; Are the implementations/APIs heavily reworked for 1.0 or is it mostly what I’ve been seeing in current gallery repo?
Yes it’s been heavily reworked and there will be breaking changes
&gt;  What I did above was from sentence_transformers (built on top of transformers) as provided by huggingface, it’s not model hub, is that ok?
Yes it is ok, one of the main changes were separating model management and Bento management in the new version. We will be able to have separate API for importing existing models.

1. That’s right - in 0.13.x, the default number of workers will always be 1.  You can changed it with container arg `--workers` or set env var `BENTOML_GUNICORN_WORKERS`

2. that’s right, those are the default values and usually there’s no need changing them unless you want to fine-tune the performance

3) yes - as mentioned above, it can be changed via CLI/Container run arguments, or an env var

have you tried increasing the `--timeout` value? e.g. to set a 5 mins timeout: `bentoml serve-gunicorn … --timeout=60*5`?

That’s right, only loaded and initialized once in each worker

BentoML does server-side dynamic batching - so in the case where client side batching is not available, BentoML has you covered. And it automatically adjust wait time and batch size for optimal performance :smile:

스크린샷 2021-11-25 오전 4.50.07.png

<@UNZ8G0VV4> and thank you for your support along the way!

I assume this is running bentoml inside a docker container?

yes , basically pushing the image created by bentoml containerize to ecr and then using that in gpu enabled eks pods running gpu optimized aws ami

according to nvidia container toolkit, I need to have cuda drivers both on the base vm running these pods and the containers themselves too

check <http://github.com/NVIDIA/nvidia-docker|github.com/NVIDIA/nvidia-docker> and because I'm running bentoml as base image I'm unable to use this runtime or install drivers in container , would be great if you could help me out

Hi, I’m having some trouble using the MultiFileInput or the FileInput Adapters when posting multiple files. It only seems to ever pickup the first file sent and not the others. Is there something in particular I need to do?

To answer my own question, `logging.file.enabled=False`  is the way to go. For context, I had to do this since the volume of prediction logs was causing disk pressure - e.g. service that vectorize an input will log the vectors

<@U02KTA8M12N> "The versions were actually installed", are those installed on the system or part of the output on the bento's requirements.txt?

I think with your latest change on the `pip_packages` option, bentoml will just use the versions that's available on your system.  Can you set the packages to `torch=1.6.0 and torchvision=0.7.0` and see ?

*if I do not pin the versions in the python file then* bentoml will output
```bentoml==0.13.1
imageio
torch==1.10.0
botocore==1.23.20
torchvision==0.11.0```
and docker build will fail with the dependency conflicts


*if I pin the versions via `@bentoml.env(pip_packages=["torch==1.6.0", "torchvision==0.7.0"])` then*
I get some dependency conflict again (I don’t have the exact error at the moment but this case doesn’t convey anything about the issue I am describing either)


*if I pin the versions via `@bentoml.env(pip_packages=["torch&gt;=1.6.0", "torchvision&gt;=0.7.0"])` then*
bentoml will output
```bentoml==0.13.1
imageio
torch&gt;=1.6.0
botocore==1.23.20
torchvision&gt;=0.7.0```
and docker will successfully build the image with the following versions installed
```torch==1.10.0
torchvision==0.11.1```
 -----
I am fine pinning the version but I feel it is a bug that if I leave the dependency version selection up to bentoml it will select incompatible/conflicting versions

<@U02KTA8M12N> Yeah I think this is something we should improve on a lot. With the 1.0 version it will be more straight forward and remove the guessing game like this from the current version.

Hi <!channel>, the much anticipated 1.0 release is around the corner and we are hosting a community demo next Thursday(12/16) at 5PM PST, to talk about some new features in the upcoming 1.0 version. We will discuss the following topics this time and would love to see you there!

• New model packaging &amp; development workflow
• Pipelined execution via Runners
• Server-side batching
• Async support
If you haven’t already, please signed up here to get beta access and an invite to the meeting <https://forms.gle/8t8gni2w5mAdFeW37>

Cheers!

<!channel> We are hosting our community demo and feedback for the upcoming BentoML 1.0 release in an hour.  Please DM me if you want to be part of the zoom call, but haven't sign up yet.  Looking forward to seeing everyone soon

not yet, I will put together a release note when the release is ready

:heavy_plus_sign: I was there and I would still love to see the recoding and/or the slide deck. :smile:

you can install bentoml 1.0 with `pip install bentoml --pre`

oh interesting. is there a set of docs w 1.0 pre?

I don't think we have webhooks right now, unfortunately.

Hi All, I know many of us, including myself, are searching for the BentoML 1.0 Preview video. I managed to get a URL from the community - <https://www.youtube.com/watch?v=r6t00Rq1MDY>

Hey <@U02RB7RF8UF>  I don't think this notebook is up to date. You can find the QuickStart guide for the latest 1.0 at <https://github.com/bentoml/gallery/tree/main/quickstart|https://github.com/bentoml/gallery/tree/main/quickstart>

We are updating the gallery with new examples to show case the new possibilities 

Yes as <@UKH494PHP>shared, the `main` branch in gallery is being updated for the 1.0 release, keep in mind that 1.0 is still under active development and 1.0 will bring lots of breaking changes to the library. I definitely recommend going through the new quickstart guide and core concept pages even if you are very familiar with previous versions

image.png

That is right, `bento save` workflow is completely changed, the new workflow requires writing a `bento.yaml` file, and calling either `bentoml build` CLI command or the `bentoml.build(…)` python API

Looks like you are still using the 0.13.x branch since it used `bert_encoder_service.save`

Screen Shot 2021-12-29 at 1.57.02 PM.png

Hi Yilun, it actually means the GPU 1 and 2 will be assigned to `model_b` runner, the Runner’s implementation will decide how many workers will be started - it is customizable in the Runner’s implementation

We are still working on documentation for 1.0, however you can find an example of this from the built-in Runner implementation for XGBoost model here:

```    def num_concurrency_per_replica(self) -&gt; int:
        if self.resource_quota.on_gpu:
            return 1
        nthreads = self._booster_params.get("nthread", -1)  # type: int
        if nthreads == -1:
            return int(round(self.resource_quota.cpu))
        return nthreads


    @property
    def num_replica(self) -&gt; int:
        if self.resource_quota.on_gpu:
            return len(self.resource_quota.gpus)
        return 1```
<https://github.com/bentoml/BentoML/blob/main/bentoml/_internal/frameworks/xgboost.py#L209-L221>

Thanks Chadoyu! (Apology for late reply, I didn’t have notification for this slack channel). I guess my main question would be:
&gt; will I still need to create multiple bentos (of the same model) in order to host on multiple GPUs like  I was doing in pre-1.0 era?
Since this will largely reduce the complexity of the whole pipeline for me.

&gt; will I still need to create multiple bentos (of the same model) in order to host on multiple GPUs like  I was doing in pre-1.0 era?
<@U01S7CRK58B> the answer is no, you dont’ need to do that with BentoML 1.0

both for single machine with multiple GPU and GPU cluster, BentoML and yatai on kubernetes will handle that for the user

Need to ramp my head around bentoml 1.0 then.

The old docs are still available at <https://docs.bentoml.org/en/0.13-lts/api/adapters.html#>

I know you can load a MLflow model from a `run_id`  (<https://github.com/bentoml/gallery/blob/0.13-LTS/bentomlflow/mlflow-to-bentoml-example.ipynb|as shown in this notebook>) but as I’m working with NLP models a lot, I tend to pack everything up into a pyfunc (model + tokenizer) and can simply load the model using `mlflow.pyfunc.load_model` and its `.predict()` <https://www.mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#mlflow.pyfunc.PyFuncModel.predict|method>.

It would be ideal to be able to wrap these pyfunc models around a BentoML service and use it for deployment.

Technically I can create a custom MLflow artifact and service extracting the `.predict()` from the pyfunc but this is not ideal cause <https://stackoverflow.com/a/65888860|pyfunc forces you to define your predict that way>
```class MyModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        # load your artifacts
    def predict(self, context, model_input):
        return my_predict(model_input.values)```
and that `context` parameter is messing up with the `predict()` method I need for BentoML

PyFunc support is already supported in the 1.0, we are still working on related documentation tho

Btw here’s related inline documentation on how to work with MLFlow saved models as PyFunc flavor in BentoML <https://github.com/bentoml/BentoML/blob/d55741b5932809f9ad8698eeaea5ab9f9abe3c22/bentoml/_internal/frameworks/mlflow.py#L86-L150>

thanks! I found it and started playing around with the beta version

I found an issue with the loading method that I just filed <https://github.com/bentoml/BentoML/issues/2160|here in github>

I’m curious though

if I’m saving a pyfunc model with such structure
```class ModelPyfunc(mlflow.pyfunc.PythonModel):
    
    def load_context(self, context):
        self.model = clf
    
    def predict(self, context, model_input):
        return self.model.predict(model_input) ```
how can I create a bento service around it? the model itself already contains the predict method, so not sure what would be a working equivalent to <https://github.com/bentoml/gallery/blob/main/quickstart/iris_classifier.py|this in your quickstart>

You can import the entire PyFunc model as a BentoML model:

```bentoml.mlflow.import_from_uri(...)```
And then load the PyFunc model back as a Runner and use it in your Service definition:

```runner = bentoml.mlflow.load_runner("name:version")

my_svc = bentoml.Service('my_service', runners=[runner])

@my_svc.api(...)
def predict(...):
    runner.run( .. )```

Basically the `runner.run` will be sending input data to the `predict` function in your PyFunc model definition

and it will run in its own process, and input will be batched

<@UKB4CLKP1> 
It seems the sign up link is wrong. Please check again. 

ah thank you <@UNZ8G0VV4> <@U02V1DRMW65>, I’ve just fixed the link

Hmm…I am still getting file does not exist error :thinking_face:

<@U018VA5QD3N> could you help check again?

<@UKB4CLKP1> Brilliant progress! Any chance we can record this for people not in US time zone?

Is there a recording of the community meeting? That would be great!

Yess, <@U02UXRR9TGU> here is it <https://www.youtube.com/watch?v=r6t00Rq1MDY&amp;lc=UgxbiuX9ID8iQypWb7h4AaABAg>