BentoML

Thank you everyone for coming to our event today, we really appreciate all the feedback!

Hello. we recorded the demo and is editing, will post up on youtube. I will be sure to notify you.

The bentoml 1.0 community video is currently unlisted. we will mark content listed soon.

For now, if you want to watch the 1.0 video. here is the link: <https://www.youtube.com/watch?v=r6t00Rq1MDY>

here’s the recording from the yatai meeting last week:  <https://youtu.be/ymcl4Pis8EM>

<@UKB4CLKP1>... thanks for that.  Just watched.  Very informative. For running up locally, rather than a minikube, I'm using a local tanzu-community-edition (TCE) cluster, which is built on Kind. I assume that works as well a a minikube?  In the helm, is there a 'minimal' workable deployment possible at the outset?  Also, versus a full-blown yatai deployment, is it possible to have a bentoml model encapsulated for running on knative (ie. in a serverless way)?  Knative is bundled as part of a TCE cluster. Just trying to understand the different possibilities. Tks

<@U0312L6442Y> yatai mostly uses standard Kubernetes API, so it should work on TCE cluster - although we haven’t tested in that environment, would love to hear how it goes with you setting

What do you mean “a minimal workable deployment possible”?

And yes it is possible to deploy BentoML directly with knative, without Yatai. However you may miss out some very useful deployment features in Yatai

hey :slightly_smiling_face: could you share something with me when you have some results. Sorry that I don’t have anything to share but I’m really interested in learning more

Screen Shot 2022-02-17 at 12.25.02 PM.png

Hi <@U0326RE5J3A> <@U03161Q7DPT> - there isn’t an example project for this yet, but we will definitely build a gallery example project for it soon.  Scaling up inference graph is actually a key feature in BentoML 1.0, attached is an example we showed in our 1.0 community demo meeting:

Under the hood, on a single node deployment, BentoML will be able to utilize all CPU cores/GPUs via multiple python processes, Runner (or each step in the inference graph) will be schedule on different processes, to run in parallel

In a distributed deployment on Kubernetes created with Yatai <https://github.com/bentoml/Yatai>, each step will be scheduled on their own Pods/ReplicaSet, and scale individually - this will avoid bottle neck in your inference graph pipeline, and maximize your overall resource utilization.

 is this what you are looking for? <https://github.com/bentoml/BentoML/blob/main/bentoml/_internal/server/__init__.py#L46|https://github.com/bentoml/BentoML/blob/main/bentoml/_internal/server/__init__.py#L46> it is currently an internal API tho so internal changes may break your usage in future versions. It is definitely possible for us to make it a user facing api. Do you mind share a bit mode why calling it programmatically can useful and how you plan to use it?

yes thanks a lot, how is the right way to import this without using _internal ?

you can directly import it with _internal in the import path: from bentoml._internal.serve.server import serve_development   it just means this API was internal and not meant to be used by users. We want to make sure our public APIs are stable and users know it for sure when an internal API is used 

Ah I think that's an internal API that's no longer exist, yes it has been replaced by bentoml.build, We will update the docs 

For those interested in this issue, a solution has been posted in the discussion thread here: <https://github.com/bentoml/BentoML/discussions/2282>

image.png

Great work <@U033GCVH3J4>! Love this illustration a lot!

It is well written. Thanks for sharing. I am going to adopt it for my next hands-on session and Yes, I am going to talk about BentoML. :slightly_smiling_face:

Hi <@U033QDDCYRZ> - do you mind tell us a bit more about your use case? Is this runner only gonna be used once?  Is the goal to serve many models on the same server and only one model is used at a time, so you don’t want to keep all the models in memory?

Hello Chaoyu, please to meet you and thanks for your answer.

Yes, our goal is to serve many random forrest models (60) with the same service and we don't want to load all models in memory at start, peraphs mantaining the most recents in memory in a sort of "cache"

I took a look at bentoml's code and it seems that the model is loaded to memory when doing .run() on a runner. Therefore I was asking if one can do a sort of unload or shutdown of a runner, Thanks

Has anyone seen an issue on AWS, where a bento works locally, it RECIEVES http traffic on an EC2 instance , but somehow there is content filtering where http posts to predict always return empty images?

Like i get a post to my predict enpoint. Locally that post has image data. The same post to the AWS instance has empty image data. WTF lol.

<@U02937SRW0Y> what's the command/tool that you're using to post?

```
aws ecr get-login --region us-east-1 --no-include-email

#ensure we build the latest bento service via
cd app;
python serialize_service.py
cd ../

saved_path=$(bentoml get myservice:latest --print-location --quiet)
sudo docker build --tag=<http://xyz.amazonaws.com/ecr-repo|xyz.amazonaws.com/ecr-repo> $saved_path```

then I update task definition and update the ECS cluster

it bounces the task and we are (usually) good to go

our EC2 instance had a hardware failure though - so it bounced, the auto scaling group spun up a new one

and now i cant post images to the end point

Hey Vade, good to see you again. Happy monday too :laughing:

Has this deployment worked before, or this is something new?

its just a recent build that has the issur

honestly i doesnt appear to be bento relatedbecause a bento serve myservice:latest works locally

so im sort of grasping at straws as to what is different on AWS side :disappointed:

I see.  we are working on new project called bentoctl.  It can help you deploy to cloud services quickly.  Maybe we can hop on a call and see if it is something you need

Ah cool. weve actually been losely following it

TBH we need to

• adopt bento 1.0 properly - we have an internal python package we for our longer task based (non realtime) services which Bento should use rather than a raw ONNX model for example. We want use 1.0 for the runner features.
• weve been aware of bentoctl and have been sort of looking to learn more it tbh - so id be curious to check it out. It might make sense to explain a bit our deloyment situation because its def NOT standard

I can see when our actual dev ops guy is around if he has bandwidth to join on a call?

Ha. Turns out we werent pinning our dependencies correctly. Now that we are issue is resolved. Whew!

Glad it got resolved.  let's grab a time to chat about your infra

Hi I'm interesting in knowing how to handle both headers and files in the latest pre release of BentoML

hi <@U02UXRR9TGU>, great question! What you are trying to build with Yatai is definitely permitted by the license. I think the elastic license v2 faq doc here does a great job explaining it: <https://www.elastic.co/licensing/elastic-license/faq|https://www.elastic.co/licensing/elastic-license/faq>

Basically the same applies for Yatai:

“Can you provide some examples around what qualifies as "providing the software to third parties as a hosted or managed Service" or not?
I'm using Elasticsearch to put a search box on my cat-picture SaaS product.
This is permitted under ELv2. Meow!

I'm a contractor setting up Elasticsearch and Kibana for my clients to use internally.
This is permitted under ELv2, because you are not providing the software as a managed service.

My cat-picture SaaS product shows view-only Kibana dashboards of analytics on searches and views.
This is permitted under ELv2. The use of Kibana in this case is limited and this does not represent access to a substantial portion of the functionality of Kibana.

I am a Managed Service Provider (MSP) running Elasticsearch and Kibana for my customers.
If your customers do not access Elasticsearch and Kibana, this is permitted under ELv2. If your customers do have access to substantial portions of the functionality of either Elasticsearch and Kibana as part of your service, this may not be permitted.

I provide Elasticsearch and Kibana as a service, where my customers have direct access to substantial portions of the Elasticsearch APIs and Kibana UI. 
This use is not permitted under the ELv2. Please reach out to us to discuss your options.”

Thank you so much <@UKB4CLKP1> for the details. We'll take that into account when making our decision.

Hi Erick, Yatai is actually built for standard Kubernetes, not for any specific provider. You can use Digital Ocean’s managed kubernetes service to run Yatai today

We're using K8S on DO for all of our deployments and we'll like to have our ML services there too.  Is there any plans to support other providers down the road?

I was looking the docs and I'm only seeing production setup for AWS.

The default Helm chart can install everything you need to run Yatai in the kubernetes cluster. There’s an option to use your own postgreSQL database for metadata storage or your own S3 bucket or Minio instance for storing model/bento files.  The docs currently shows how a user can configure those in an AWS environment, but you can do something similar with digital ocean as well, for example use their managed database, or manage Minio on DO separately(<https://www.digitalocean.com/community/tutorials/how-to-set-up-an-object-storage-server-using-minio-on-ubuntu-18-04>)

that’s good to know! If it’s fully s3-compatible, it should work with Yatai in theory, our team haven’t tested it yet

would love to hear how it goes if your team want to give that a try

We'll see if we can come up with a PoC using DO and if everything goes smoothly (fingers crossed, hehe) we'll make a PR for anyone  wanting to use DO instead of AWS.

That'd be really helpful for the community!

Hello   I am exploring BentoML to create service from my model. As a begineer, I tried few of the examples given on Github. I am getting below error:

``` ---------------------------------------------------------------------------```
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-24-5efba0a644ea&gt; in &lt;module&gt;
----&gt; 1 bentoml.sklearn.save("iris_clf", clf)

AttributeError: module 'bentoml' has no attribute 'sklearn'

```AttributeError                            Traceback (most recent call last)
&lt;ipython-input-23-cfd1046a86b2&gt; in &lt;module&gt;
----&gt; 1 sklearn.save("iris_clf", clf)
 
AttributeError: module 'bentoml.frameworks.sklearn' has no attribute 'save'```


I also tried to downgrade the version.  Current version is BentoML==0.13.0

But so far, not able to figure out. What else can I do? Could someone please guide me?