BentoML

<@UKB4CLKP1> is presenting BentoML at Kubeflow's community meeting today. if you are interested you can join it at <https://zoom.us/j/799749911>.  It starts at 5:30 PM PST

Just shipped something you'll all find handy! :slightly_smiling_face: Highly recommend this free chrome/firefox extension as a must-have. It automatically finds code implementations for machine learning papers anywhere on the web (Google, Arxiv, Twitter, Scholar, and other sites)

<https://chrome.google.com/webstore/detail/mlai-code-implementation/aikkeehnlfpamidigaffhfmgbkdeheil>
or
ht<tps://addons.mozilla.org/en-US/firefox/addon/code-finder-catalyzex/>

hi <@U018Y52EH1N> did you install bentoml from source?

Do you mind start a thread in GitHub discussions?

Hu <@UKB4CLKP1>, I'm installed from pip. Now i had deploy a yatai service on different machine and set yatai endpoint in bentoml configuration. In this way work

No only here, because I thinked that is my mistake and i don't open a issue on github

Hey <@U018Y52EH1N>, is it like the tab title says "Bento ML" and the page is full blank ?

No, it's a uanble load page, if I use telnet i have a Connection refused

I thought “latest” is just a shortcut generated automatically?

i think you manually need to tag and release a 'latest'

Ah I see, I always thought “latest” is dynamically generated when there’s new image - do you think we should add tag “latest” tho?

Latest is handy in dev environment but seems like a bad idea when it comes to production deployment

from what i've seen, most popular images do manually release a latest tag though

i think it would just be an extra line in the release script to also tag it with latest before pushing

Hi <@U0195CF5QH2> - looks like there are some issue with this saved bundle

could you tell me more about how you got here?

So I uploaded my file  "20200825112818_29410B" on github  after succesfully  testing it locally...The Image got built..But when running it had some issues ..

this was part of my tekton CI/CD

Did you test building a docker image with 20200825112818_29410B directory locallly as well?

```[2020-08-26 14:08:16,263] INFO - Starting BentoML API server in production mode..
[2020-08-26 14:08:16,720] INFO - get_gunicorn_num_of_workers: 5, calculated by cpu count
[2020-08-26 14:08:16 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-08-26 14:08:16 +0000] [1] [INFO] Listening at: <http://0.0.0.0:5000>  (1)
[2020-08-26 14:08:16 +0000] [1] [INFO] Using worker: sync
[2020-08-26 14:08:16 +0000] [17] [INFO] Booting worker with pid: 17
[2020-08-26 14:08:16 +0000] [18] [INFO] Booting worker with pid: 18
[2020-08-26 14:08:16 +0000] [19] [INFO] Booting worker with pid: 19
[2020-08-26 14:08:16 +0000] [20] [INFO] Booting worker with pid: 20
[2020-08-26 14:08:16 +0000] [21] [INFO] Booting worker with pid: 21
[2020-08-26 14:08:17,083] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.8, but current environment version is 3.7.6.
[2020-08-26 14:08:17 +0000] [17] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/opt/conda/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/opt/conda/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/conda/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/conda/lib/python3.7/site-packages/bentoml/server/gunicorn_server.py", line 91, in load
    bento_service = load(self.bento_service_bundle_path)
  File "/opt/conda/lib/python3.7/site-packages/bentoml/saved_bundle/loader.py", line 199, in load
    svc_cls = load_bento_service_class(bundle_path)
  File "/opt/conda/lib/python3.7/site-packages/bentoml/saved_bundle/loader.py", line 119, in load_bento_service_class
    metadata["module_file"], bundle_path
bentoml.exceptions.BentoMLException: Can not locate module_file holt_classifier.py in saved bundle /bento
[2020-08-26 14:08:17 +0000] [17] [INFO] Worker exiting (pid: 17)```
This  is the error I keep getting <@UKB4CLKP1>

Do you mind share the entire bundle file? If it doesn’t contain sensitive data

Or run tree command to show me all the files and structure in the bundle?

after seeing the github issue I think I understand what’s going on now 

You created the bundle on a windows machine and deploy on Linux right?

ummm not exactly <@UKB4CLKP1> ...It was the same OS

<@U0195CF5QH2> What is the python version?  could you also share the content of bentoml.yml file?

bentoml.yml

<@U0195CF5QH2> you have tried building and runing the docker image with this saved bundle and it all works on your local dev machine right?

It is only a problem when running on tecton?

could you share a bit more detail so we can reproduce it?

My teammate <@U019Z0NRW81> will explain it

we see errors while running the pipeline over tekton however when we try to deploy it using local system it goes through

this is the error we see continuously -
```metadata["module_file"], bundle_path
bentoml.exceptions.BentoMLException: Can not locate module_file holt_classifier.py in saved bundle /bento```


Hi Soumo, could you tell me how did you set up the tekto pipeline 

Step by step instructions that can allow me to reproduce on my end would be most helpful 

did you use docker to deploy? How was the docker image created? Were you testing the same image on local system?

Step by step instructions that allows me to reproduce would be the best 

 can you try reproduce it with a local tekton &amp; Minikube deployment using the BentoML QuickStart example?

Would be great if you could create a GitHub bug report issue and fill out the template 

yes, but lets have a working session with you...may be we are missing something

It's a feature of `cp`. <https://linux.die.net/man/1/cp>
When /bento doesn't exist, `cp . /bento` means copy the current directory as the new directory `/bento`

This project will definitely continue to be open sourced, think of the relationship as Databricks to Apache Spark, and Datastax to Cassandra

If your team are serious about adopting BentoML, we’d welcome you to join the core team and get a saying in where the project goes. You may reference the governance structure of this project <https://github.com/bentoml/BentoML/blob/master/GOVERNANCE.md>  We definitely welcome people from other organizations to get involved with the project and help the open source community to grow. If your team could shows commitment to the project(PRs, feature requests, community involvements) and expertise in the space, I’d be happy to invite you to join the core team

Of course, you always have the option to fork it and maintain it by yourself. The project is under Apache License 2, so forking it and keeping your fork open source is not a problem.  But eventually you may lose the benefit of having a community around the project.

We are an early stage startup in SF, adopting the Open Core business model(<https://en.wikipedia.org/wiki/Open-core_model>). The key to our company’s success is to have a successful open source project that truly provides value to its community. That being said, the project will definitely remain open source, and we will continue to build features and support the community.

The BentoML project is not our solo product. We are building a fully managed YataiService for organizations to easily manage prediction services created by Data Scientists. (Think of it as Docker hub to docker developers).

Hope that make sense and addresses your concern. Happy to chat more if you have any other questions or concerns.

Thank you for the detailed and thoughtful answer. I will look into this more.

I have something like this. You have to change `localhost`. In my case, I have to change it to `docker0` IP address on linxu. On mac, it is `host.docker.internal`.

```scrape_configs:
  - job_name: 'bentoml'
    scrape_interval: 20s
    static_configs:
      - targets: ['localhost:5000']
        labels:
          instance: 'bentoml:5000'
          job: 'titanic'```

Hi Keith, yes it is on GitHub issues. Are you looking for MLlib support? Or batch offline inference with other ml frameworks on Spark?

For MLlib support there’s an open PR for it, but need someone to take over the PR

Currently looking at both MLlib and batch, as we will eventually need to support both. I cannot take the MLlib PR right now, but may be able to once we have other stages set up and existing frameworks running

Awesome, let me know when you’d be available to help with MLLib support.  As for batch inferencing on Apache Spark, it is currently doable via BentoML’s python API, although it requires some manual work. Let me know if that is a blocker for you, happy to provide some quick workaround on that. And we are planning to make it easier to deploy to Spark after the ongoing input adapter refactoring: <https://github.com/bentoml/BentoML/issues/1002>

Hi <@U019R38D1GW>, glad you asked about it, I’m planning to use the Github Discussions to share our current roadmap with priority, timeline, need help etc. For now, you can take a look at the existing feature roadmap here:   <https://github.com/bentoml/BentoML/issues?q=is%3Aopen+is%3Aissue+label%3A%22new+feature%22>   

I am using MLFlow for managing the experiments and exploring BentoML for serving. Could you provide me any directions?

I don't know if that's the recommended way, but what I sometimes do is train a bento service together with the experiments, and log its name with MLFlow, so when I decide on the best model I already have a service waiting to be deployed.

<@U01AKGQ5CF3> I’d recommend the approach <@USGHW96R0> shared above. You can also build a second pipeline that pulls trained model from MLFlow and then package it with BentoML for serving. Although managing two separate pipeline takes more time and could introduce more issues.

bento_serve.png

could you try add `@env(pip_dependencies=['statsmodels'], auto_pip_dependencies=True)`  ?

BentoML can not figure out dependencies that are only used in the pickeled object

bento.png

Nope <@UKB4CLKP1>  ..I had to add statsmodels in my requirements.txt manually  after the file got generated

`@env(auto_pip_dependencies=["statsmodels == 0.10.1","joblib","pickle","numpy"]`


For some strange reason when I do this it works

auto_pip_dependencies is suppose to be a boolean value

Also  is there a way to add statsmodels to the requirements.txt from the code?

this should add it to the requirements.txt

what’s the error when you do `@env(pip_dependencies=['statsmodels'], auto_pip_dependencies=True)`  ?

Hey <@UKB4CLKP1>, Sorry for the late reply as I wasn't working over the weekend

bento_eror.png

I'm attaching the code and error in this picture

It works fine when I add statsmodels==0.10.1 in req.txt

I see, try changing the code to `pip_dependencies=['statsmodels==0.10.1']`  ?

The next release of bentoml will do a better job at parsing the pip package version requirement and handle this case

but in current release, the extra space will result in that error I believe

<@UKB4CLKP1> Sure..wil try it out and  come back to you

<@UKB4CLKP1> still facing the same  issue unfortunately

could you verify the content of `requirement.txt` ?

Have you tried removing it from conda_dpendencies?

<@UKB4CLKP1> requirements.txt doesn't get updated still..will try removing it from conda_dependencies

tried removing it from conda_dependencies..Still the issue unfortunately

The good news is now it runs through CI/CD

<@U0195CF5QH2> great to hear.  I can not reproduce the dependency issue on my end tho. If you specify `@env(pip_dependencies=['statsmodels==0.10.1'])` , then the `requirement.txt`  file is expected to have the ``statsmodels==0.10.1`` line.

could you create a simple BentoService class in your env and help me to verify this?

it worked!! I removed the auto_pip_dependencies=True part and now it works really well..probably it was overriding it....

<@UKB4CLKP1> Can I contribute a simple example to the bentoML docs ?? so that next time someone wants to deploy a time-series statistical model they can just run a notebook I create?

yes definitely! Thank you! I should have fixed the issue with overriding, the fix will be out in the next release 

thanks <@UKB4CLKP1>..Will raise a PR tomorrow

Any procedure to contribute?? Like where to upload the notebook

Do I just upload my code <https://github.com/bentoml/gallery|HERE> and upload my notebook on a new branch and add it to the readme.md ??

you can create a new folder for statsmodel

if you can take a look at other example notebooks and follow the same format that’d be great

and you can also add it to the readme in the main repo, after the example notebook is added

<@UKB4CLKP1> np :slightly_smiling_face: ..I can follow the irisclassifier document  kinda sequence right?

bentoML.png

I can create a new section right,named statsmodels? <@UKB4CLKP1>

you can take a  look at other gallery examples

Hey <@UKB4CLKP1>..Are you planning to add it in the bentoml docs as well?

<@U0195CF5QH2> I thought you already did?

btw, how’s your deployment with BentoML going?  Have you pushed everything to running in production yet?

<@UKB4CLKP1> Sorry,  I just did..created a new PR..Sorry for the trouble

<@UKB4CLKP1> me and my manager are currently comparing the options at hand..The current project is just an incubation one ...Maybe by October second week we'll have an answer

Thanks <@U0195CF5QH2>, let me know if there’s any thing we could help

Hi there! I'm new to BentoML: it looks really cool and I'm looking forward to using it.

Yes. <http://BentoML.ai|BentoML.ai> is a company that started the BentoML project. The relationship is similar to Databricks to Apache Spark, and Datastax to Cassandra

So you (are planning to) built more advanced/user-friendly/enterprisy stuff on top of BentoML?

Anyway, I thought the project and especially the documentation is looking very good!

Yes, we plan offer commercial product in the future.  There was a great thread that Chaoyu talked about the company and future plans.  <https://bentoml.slack.com/archives/CK8PQU2JY/p1598648820019600>.

Open-source is our core. The company's success is depended on to have a successful open source project that truly provides value to our community.

<@U01A8FDTSK0> yes we are building advanced enterprise features on top of the open source BentoML, is that something your team may be interested in? happy to chat more

Thanks! Those explanations are really helpful!

We just decided to use BentoML to wrap our models for deployment, and also be able to do offline testing with the exact same code, including pre- and post-processing.

That’s awesome, let me know how it goes! Would love to hear more of your feedback and thoughts

Hi all! Colleague of Pieter here. Pieter started this project at out company, but is currently on holiday and I am taking over for a bit. It looks really awesome! :smile:

Our manager however just asked some critical questions on how BentoML compares to the Google Cloud AI Platform. My knowledge about both the AI platform and about BentoML is not extensive enough yet to make this comparison and argue for one or another. My intuition from what I've learned so far would be that BentoML supports more ML frameworks and more diverse deployment options. Do you have any input? :slightly_smiling_face:

Hi <@U01C3RNJZ5E> - I actually just had a call with <@U018VA5QD3N> about their experience migrating from Google Cloud AI platform to BentoML.   In Shihgian’s words: Google Cloud API Platform has a strong opinion on how you architect your applications which introduces fragmented architecture and frictions to DevOps and Engineers who are operating those ML workloads in production. Whereas BentoML is highly modular and gives DevOps &amp; engineers the flexibility to deploy in their own infrastructure, and integrate with existing infra tooling for monitoring, log collection, tracing, etc.

And yes you are right, in terms of features, BentoML supports more ML frameworks and more diverse deployment options!

To add to Chaoyu’s comment, Google AI Platform’s first class ML library is TensorFlow. The sklearn version is very far behind. So, we could not use the predict probability function on a later version of sklearn for one of our ML algorithms. We had to copy the source from sklearn site and include it in Google AI platform. Google AI platform will improve sklearn deployment later but it is not their primary focus. We were very frustrated with Google AI Platform.

Thank you guys! I really appreciate the quick response. This aligns with what I've read so far about the different platforms. I'm with Pieter! BentoML seems like the way to go! Thanks again! :+1:

I experimented with most of the AI pipeline solutions out there, days of deployment and tests to revamp our existing, painful and slow existing SDLC process. We wanted to be able to ship models to production faster and throw experimental, canary instances to evaluate their prediction quickly, against production data.

BentoML is by far the most well thought design of all. It doesn't get in the way yet does what it's meant to do well. It's also the most robust/versatile. Kubeflow/Kserving were promising but ironically can't even be deployed on the latest version of k8s.

I was able to create a build + deploy pipeline to our clusters, with canary instances in about a day or so.
I had to come in here and say thanks!

Thank you for sharing, Hirako!  Really glad to hear about it!

We see the pain points in adopting Kubeflow/KFserving for model deployment and we are actually building a model orchestration layer for bentoML on Kubernetes. I will keep it posted on our progress here in the community.

it's rather easy to orchestrate as it stands. the Networking aspect works well with existing solutions e.g knative/istio.

in the roadmap, I see:
_Advanced model deployment workflows for Kubernetes, including auto-scaling, scale-to-zero, A/B testing, canary deployment, and *multi-armed-bandit*_

The first _4_ features are not pain points. But the last one is interesting. Right now we have to build our own strategy.

Thanks for the feedback <@U01BFSC8N91>  may I ask how do you currently do A/B testing and Canary deployment? Was it operated by a DevOps team?

we didn't have A/B testing/canary deployment until now. Now using knative with its revision system.

in our case, the data science team is the devops team. we don't have dedicated deveops roles. Of course not everyone is looking at the devops aspect.

I see that both predictions and feedbacks are printed in log files on the filesystem. Is there a consideration to rather send those, via configuration, to a centralised store?

The same way some network metrics are gathered by prometheus, there is value in gathering predictions (outcome of the models) and feedbacks (income to the models) to an aggregator for analysis. Right now we would have to parse the logs on X number of servers.

Hi Hirako, log collection can be challenging when working with high volume prediction services, that’s why we left it out-of-scope for BentoML, and makes it easy to work with log collection tools like logstash and fluentd.  Although those tools are not ideal for data scientist to access and use those data for analytics or model development.

In my opinion, this is something very hard to offer as an open source component that’s applicable for everyone, but could be part of our commercial product (a model deployment lifecycle management platform on top of BentoML)

OK. currently using fluentbit (similar to fluentd).  Prediction and feedback is not really logging though.

if I follow your logic (dedicating the centralized info to things like fluentd), then the predictions should be printed to a different appender. the same way feedback entries are appended to a feedback specific appender.

with that, we can have fluentd or other aggregator taking care of it. as it stands it would aggregate all prediction outcomes along with the rest of the server logging. (irrelevant for analysis of predictions).

Hi. You can find the instruction in the docs of ImageInput

Using low level FileInput and call pil.Image.open by youself to get the Image object

<@UQF2L5GQ0> so you must resize in bentoml.BentoService, don't you?

So this is a question about tensorflow? Would you like to share your attempts and the error stack output you encountered?

You can resize data inside or outside of the temsorflow gragh.

i try to resize, then trow th following message :
 Exception happened in API function: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'read'

How about just <http://artifact.model.to|artifact.model.to>(device) ?

hi Mayur, thank you for your response. I do also thought about using <http://torch.nn.Module.to|torch.nn.Module.to>() function but i have to put the same logic for all of my endpoint definitions in the bentoml service. So its not that convenient. If there is a function that called on startup it will be great to put it there i guess.

Thank you, this is a good suggestion. We are already considering adding post and pre start hooks.

haha, welcome. feel free to ask questions, happy to answer

Hey, everyone. I'm Harshit. I'm a 4th year, CSE student. I've currently enrolled in MLH Fellowship open source program. I've been assigned Bentoml projects and will be contributing to Bentoml.

Looking forward to contributing, having a good discussion, and taking lots of help from everyone.

If anyone has any tips, how to get started, or anything in general, feel free to text me.

I'll be looking in the documentation, reading a few things in BentoML repository, for the time being, If I get stuck, I'll surely reach out for help. Hoping to get comfortable quickly and help his beautiful community the best I could.

Hey everyone,
I am *Aditya Raman*, MLH-Fellow and a final year student with major in Computer Science. `BentoML` has been assigned for me as a part of MLH-Fellowship. I am looking forward for making contributions.

Feel free to DM me for any resources/references or discussion about the softwares and technologies and specially `BentoML` .

Hi Marian - BentoML does not provide data drifts monitoring out of the box, it standardizes the prediction logging format and makes instrumenting your model serving workload easy. That allows users to build monitoring solution on top

BentoML does come with basic service monitoring out of the box, and gives the user API to instrument additional metrics