BentoML

Hey! We confirmed that this is a bug at the moment. We're looking into fixing it

added this issue here: <https://github.com/bentoml/BentoML/issues/3111>

Unfortunately, right now I don’t know a way to overcome it

Sure, I'll dig deeper into it. Thanks Bo!

Another thing, is there a way to generate bentoml folder in the project directory instead of the default home directory? I am referring to the folder that is created after we save the model with bentoml.

I want to upload my project directory on ec2 instance to practice rest of the bentoml course material.

Also, I like to organize everything in my project directory. Not sure if it is best practice though.

I see. 

Usually users choose to go with the default home as every time for any cli you will have to explicitly say where the bentoml home directory is getting very annoying 

Bentoml offers import and export commands for you to organize in an remote location

There is a way, though, in case you still want to do it: `BENTOML_HOME=&lt;your project directory&gt;/bentoml`

Is there an error that is printed? I know that currently because we're using the python implementations of compression it can take quite a bit for large models, especially on slow hardware.

Is your CPU working while that's happening?

<@U047RL51J9F> what's the size of your model ? (bentoml models list) and is your CPU working while this hangs?

• 197.77 KiB
• yup, my cpu still works

16gb ram
ryzen5 4600h
ubuntu 22.04.1 LTS 64-bit

The last person I helped with a hanging issue their machine was too small (1gb, 1cpu) but yours seems big enough

Do you know how much memory is allocated to docker?

Oh, I think another person had issues with access. Did you change where the bentoml home directory is?

im using virtualenv in my mlzoomcamp folder

checked it.
my bentoml folder is in `home` but using my command here
`/home/chanchan/Documents/data_projects/ml_zoomcamp/07_bentoml`

I think that should be fine. Can you try uninstalling bentoml? And then checking if you have bentoml installed in your base environment?

also, just to be clear, you need to install bentoml with pip

tried it . no luck :face_with_thermometer:

here's the bentofile fyr
```service: "service.py:svc"
labels:
   owner: bentoml-team
   stage: dev
include:
- "*.py"
python:
  packages:
  - xgboost
  - sklearn
  lock_packages: false```

This is my service file:
import numpy as np
import bentoml
from <http://bentoml.io|bentoml.io> import NumpyNdarray

model_ref = bentoml.sklearn.get("mlzoomcamp_homework:latest")

# notice the latest tag so you don't have to update after every change
model_runner = bentoml.sklearn.get("mlzoomcamp_homework:latest").to_runner()

svc = bentoml.Service("mlzoomcamp_homework", runners=[model_runner])

# input and/or output could also be JSON() or some other type
@svc.api(input=NumpyNdarray(), output=NumpyNdarray())
async def classify(input_series: np.ndarray) -&gt; np.ndarray:
    return model_runner.predict.async_run(input_series)

Solved, re-wrote service file, with code that matched class tutorial.

FWIW, that usually means you forgot to `await` an async function somewhere!

Q1: HI there, while mentioning input method (input=JSON(pydantic_model=CreditApplications ) , i'm getting "Failed to load API definition. Fetch error
Internal Server Error /docs.json"

Locust file:

`import numpy as np`
`from locust import task`
`from locust import between`
`from locust import HttpUser`

`sample = {"seniority": 3,`
 `"home": "owner",`
 `"time": 36,`
 `"age": 26,`
 `"marital": "single",`
 `"records": "no",`
 `"job": "freelance",`
 `"expenses": 35,`
 `"income": 0.0,`
 `"assets": 60000.0,`
 `"debt": 3000.0,`
 `"amount": 800,`
 `"price": 1000`
 `}`

`class CreditRiskTestUser(HttpUser):`
    `"""`
    `Usage:`
        `Start locust load testing client with:`
            `locust -H <http://localhost:3000>`
        `Open browser at <http://0.0.0.0:8089>, adjust desired number of users and spawn`
        `rate for the load test from the Web UI and start swarming.`
    `"""`

    `@task`
    `def classify(self):`
        `<http://self.client.post|self.client.post>("/classify", json=sample)`

    `wait_time = between(0.01, 2`

`from unittest import result`
`import bentoml`
`from <http://bentoml.io|bentoml.io> import JSON`

`model_ref = bentoml.xgboost.get("credit_risk_model:latest")`
`# credit_risk_model:iusktlssmgtxfshc -- latest chooses teh most recent tag`
`dv = model_ref.custom_objects['dictVectorizer']`

`model_runner = <http://model_ref.to|model_ref.to>_runner()`

`svc = bentoml.Service(`
    `'credit_risk_classifier', runners=[model_runner]`
`)`


`@svc.api(input=JSON(), output=JSON())`
`def classify(application_data):`
    `vector = dv.transform(application_data)`
    `prediction = model_runner.predict.run(vector)`
    `print(prediction)`

    `result = prediction[0]`

    `if result &gt; 0.5:`
        `return {"status": "Declined"}`
    `elif result &gt; 0.2:`
        `return {"status": 'Maybe'}`
    `else:`
        `return {"status": 'Approved'}`

Does a single request to the server work?

No <@U02HFFXSZ9D>, It is still showing 100% Failure rate

<@U02HFFXSZ9D> I was able to solve the issue with help of a mate. `bento serve --production` has to be ran simultaneously. I did that, opened another terminal and ran `locust -H <http://localhost:3000>` . It is working fine now. Thank you.

Hi <@U02HFFXSZ9D> I'm having this error when I run `bentoml containerize`. What might be the issue? Is it related to the docker File?

`$ bentoml containerize credit_risk_classifier:b7zzi7cuogagbshc`
`Usage of buildx:`
      `--builder string`
`Building docker image for Bento(tag="credit_risk_classifier:b7zzi7cuogagbshc")...`
`error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: failed to copy: httpReadSeeker: failed open: failed to do request: Get "<https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/1e/1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd/data?verify=1666711055-Cgq9iDB4klH8Xj5cpMfhujRAESI%3D>": EOF`
`Failed building docker image: Command '['docker', 'buildx', 'build', '--progress', 'auto', '--tag', 'credit_risk_classifier:b7zzi7cuogagbshc', '--file', 'env\\docker\\Dockerfile', '--load', '.']' returned non-zero exit status 1.`

That looks like a containerize issue; <@U01BMCJ29LN> any ideas?

Hi. Had to uninstall bentoml, do a docker prune, pip and conda cache clearing and reinstall bentoml. It worked fine after. There was no other fix because I tried everything prior.

I haven't rewatched the bentoml videos yet, but I did notice something that caught my attention while testing my docker container.

If I perform the locust load test on my docker image, I get some really good results, similar to what was shown in the videos.

In contrast, while I was actually following along with the videos initially, I ran the load test against the service that I had running locally with just `bentoml serve...` and it would not go past 100 users.

No such issue on the docker image though. Not sure if it was mentioned in the video that it could only be used against the docker version of the service.

On the other hand, it might just be another one of those Windows quirks.

Hi <@U0484T68A87>, great attention to detail. Using `bentoml serve`  achieves a lower concurrent user count due to two reasons.
1. `bentoml serve` without `--production` runs in a single process for easy debugging. Parallelism is limited due to Python GIL.
2. Adaptive batching is not enabled without `--production`. Adaptive batching as the course materials mentioned helps improve throughput by leveraging hardware vectorization.

Hi <@U01GMPKSG0H> thanks for the info! I’m positive that I was using the production flag in both instances though. I made sure of it. Perhaps it’s like the reload flag that doesn’t work under Windows as well?

I also tried using model instead of LogisticRegression

Code:
model =LogisticRegression(max_iter=1000, random_state=1,C=100,penalty='l2',solver='lbfgs')

model.fit(x_train, y_train)

bentoml.LogisticRegression.save_model('stroke_prediction', model)

Solution if someone else gets the error:
Since Logistic Regression is contained in the sklearn library, you should try to save your model with the following method bentoml.sklearn.*save_model*
<https://docs.bentoml.org/en/latest/reference/frameworks/sklearn.html#bentoml.sklearn.save_model>

Thanks for finding this. Earlier, I thought that I would have to switch to a RandomForestClassifier instead of the XGBoost one that I was using and ran into the same issue.

<@U0484T68A87> I think for xgboost you need to use xgboost as it is not a sklearn module.

i was using xgboost and thought i had to switch to the random forest one, but had trouble figuring out how to save that model. My xbg one is fine.

Command I am running:
bentoml serve service.py:svc --reload

Resolution: in the error is states no module names 'bento', this is from a typo. Corrected to 'from bentoml import JSON'. Service now runs.

Try without reload, we’ve been having issues with it. Also, if that doesn't work, repost this in #support. Now that the main week is over, fewer people are watching this channe

Saving the model in bento:
bentoml.sklearn.save_model('stroke_prediction', model,
                           custom_objects={
                               'dicVectorizer': dv
                           })

service.py :
import bentoml
from <http://bentoml.io|bentoml.io> import JSON

model_ref = bentoml.sklearn.get('stroke_prediction:latest')
dv = model_ref.custom_objects['dicVectorizer']

model_runner = <http://model_ref.to|model_ref.to>_runner()

svc = bentoml.Service('stroke_prediction', runners=[model_runner])

@svc.api(input= JSON() , output= JSON())
def classify(application_data):
    vector = dv.transform(application_data)
    prediciton = model_runner.predict.run(vector)

    return prediction

what I am submitting at localhost:3000:
{ "gender": "Male", "age": 55.0, "hypertension": 0, "heart_disease": 0, "ever_married": 1, "work_type": "Govt_job", "residence_type": "Urban", "avg_glucose_level": 231.15, "bmi": 22.3, "smoking_status": "never smoked", "obese": 0, "clearly_diabetes": 1 }

Forgot to update, resolved, posted in support thread

Hello!

Has any one used `@web_static_content` decorator to add the frontend side to our Bento?

Is there any resource or video?

Thank you! :saluting_face: 

This is what me y service.py looks like at the moment:
```import bentoml
from <http://bentoml.io|bentoml.io> import JSON, Image

tag = "bzphu2d342ywhxz3"
model = bentoml.fastai.get(f"kitchenware_fastai:{tag}")
runner = model.to_runner()

svc = bentoml.Service("kitchenware_classifier", runners=[runner])


@svc.api(input=Image(), output=JSON())
async def classify(image):
    prediction = await runner.predict.async_run(image)
    print(prediction)
    result = prediction[0]

    return result```

Ok, so I switched to saving it as a pytorch model and added the ImageDataLoader as a custom_object and I can now access it. Problem now is that test_dl expects a list of image file paths and not the actual image. Do I have to save the image locally and then pass it the path to it or is there a better way?

```import bentoml
from <http://bentoml.io|bentoml.io> import JSON
from <http://bentoml.io|bentoml.io> import NumpyNdarray
from <http://bentoml.io|bentoml.io> import Image
from PIL import Image as pil_image


# Getting the model from BentoML and setting up runner
model_ref = bentoml.onnx.get("kitchenware_model:latest")
model_runner = model_ref.to_runner()
svc = bentoml.Service("kitchenware_model", runners=[model_runner])

# Serving the model
@svc.api(input=Image(), output=NumpyNdarray())
def predict(img):
    img = pil_image.resize((224, 224))
    prediction = model_runner.run(img)
    return prediction```
service.py file as above,

```service: "service.py:svc"
labels:
  owner: AaishaM
include:
- "service.py"
python:
  packages:
  - onnx
  - onnxruntime
  - torch
  - Pillow```
and the bentofile.yaml here.

trace.txt

Found the <https://github.com/sayakpaul/deit-tf/tree/main/vit|model source>

Updated to be more in line with the examples. Now I'm getting:
```ValueError: Input 0 of layer "model" is incompatible with the layer: expected shape=(None, 224, 224, 3), found shape=(1, 244, 244, 3)```

Thank you guys for making this library :smile:

Hi guys, I am new to Bento ml, I am trying to create a service for my ml model, I have created the service .py script 
I ran the service with the code 
bentoml serve service:svc --reload

Here is the output 

2023-06-19T14:37:32+0100 [INFO] [cli] Environ for worker 0: set CPU thread count to 4
2023-06-19T14:37:34+0100 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "service:svc" can be accessed at <http://localhost:3000/metrics|http://localhost:3000/metrics>.
2023-06-19T14:37:41+0100 [ERROR] [cli] Could not bind 0.0.0.0:3000


I am worried about the error message I have no idea what it is, plus 