https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
🍱 1
🏁 1
i
the error message
Copy code
2022-10-21T08:16:12+0000 [INFO] [runner:credit_risk_model:1] Service loaded from Bento directory: bentoml.Service(tag="credit_risk_classifier:ixasr6srdc2tmhqa", path="/home/bentoml/bento/")
2022-10-21T08:16:12+0000 [ERROR] [runner:credit_risk_model:1] Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 650, in startup
    handler()
  File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner.py", line 230, in init_local
    self._init_local()
  File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner.py", line 221, in _init_local
    self._init(LocalRunnerRef)
  File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner.py", line 215, in _init
    runner_handle = handle_class(self)
  File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 25, in __init__
    self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/frameworks/xgboost.py", line 255, in __init__
    self.model = load_model(bento_model)
  File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/frameworks/xgboost.py", line 141, in load_model
    model.load_model(model_file)
  File "/usr/local/lib/python3.9/site-packages/xgboost/core.py", line 2294, in load_model
    _check_call(_LIB.XGBoosterLoadModel(
  File "/usr/local/lib/python3.9/site-packages/xgboost/core.py", line 246, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [08:16:12] /workspace/src/c_api/c_api.cc:911: Check failed: str[0] == '{' (b vs. {) : 
Stack trace:
  [bt] (0) /usr/local/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x55388) [0xffff64a0d388]
  [bt] (1) /usr/local/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x60150) [0xffff64a18150]
  [bt] (2) /usr/local/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(XGBoosterLoadModel+0x16c) [0xffff64a183fc]
  [bt] (3) /usr/lib/aarch64-linux-gnu/libffi.so.7(+0x6048) [0xffff73f9a048]
  [bt] (4) /usr/lib/aarch64-linux-gnu/libffi.so.7(+0x5770) [0xffff73f99770]
  [bt] (5) /usr/local/lib/python3.9/lib-dynload/_ctypes.cpython-39-aarch64-linux-gnu.so(+0x134a8) [0xffff73fc04a8]
  [bt] (6) /usr/local/lib/python3.9/lib-dynload/_ctypes.cpython-39-aarch64-linux-gnu.so(+0x1194c) [0xffff73fbe94c]
  [bt] (7) /usr/local/lib/libpython3.9.so.1.0(_PyObject_MakeTpCall+0x7c) [0xffff7fdf99fc]
  [bt] (8) /usr/local/lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x446c) [0xffff7fe614ac]



2022-10-21T08:16:12+0000 [ERROR] [runner:credit_risk_model:1] Application startup failed. Exiting
any help would be appreciated. here are the verisions I am using: XGBoost: 1.5.0 bentoml, version 1.0.7
in case anybody is able to offer some assistance, there are more details in this thread https://datatalks-club.slack.com/archives/C0288NJ5XSA/p1666386548187719?thread_ts=1666340291.904869&cid=C0288NJ5XSA
a
Hi there, where are you running this on?
i
Hi Aaron. Thanks for responding. this is the paltform
Copy code
>>> platform.platform()
'macOS-12.6-arm64-arm-64bit'
a
Oh I’m wondering how did you build the bento? I assume that you are just running this locally?
i
these are the commands I entered leading up to the problem. I did these from insdide my project folder. they all seem to execute successfully until I get to 'docker run'
Copy code
bentoml build

bentoml containerize credit_risk_classifier:wh6pnkst56pu7ct7

docker run -it --rm -p 3000:3000 credit_risk_classifier:wh6pnkst56pu7ct7
a
can you try containerize the bento with 
--platform=linux/amd64
?
Copy code
bentoml containerize ... --platform=linux/amd64
This will create a x86 container instead of an arm container.
i
I got the same error when running the docker run command. however this was at the beginning
Copy code
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
should I have specified
Copy code
--platform=linux/amd64
as part of the docker run command as well?
a
Sure, but I don’t think that is the core of the problem. what version of docker you are currently using?
can you also send the output of 
docker buildx ls
?
i
Copy code
NAME/NODE       DRIVER/ENDPOINT STATUS  PLATFORMS
desktop-linux   docker                  
  desktop-linux desktop-linux   running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
default *       docker                  
  default       default         running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
Docker version 20.10.12, build e91ed57
a
sg. Thanks for the info. Can you also try to run this workflow again with the latest xgboost? to see if whether this is a xgboost or a bentoml problem?
I can reproduce this issue. I will take a look at this.
i
what is the best way to get the latest xgboost? I ended up using this method. I remember I had a problem isntalling it with pip. https://stackoverflow.com/questions/65752489/error-when-importing-xgboost-on-apple-m1/66880524#66880524
okay thanks for taking a look. I will be busy for several hours but I will definitely check later tonight.
a
I just do 
pip install xgboost
i
are you using arm64?
a
yes, I’m currently on M1
i
i uninstalled the version of xgboost (py-xgboost) I had previously and pip installed xgboost (1.6.2, it shows up in my environment packages when i run conda list but for some reason doesn't when I just do 'xgboost --version'). then I tried it all again starting with 'bentoml build' through 'docker run'. I got the same error
a
Hi @Isaac Hurwitz, The issue lies at our dependency resolver. BentoML use 
pip-compile
to lock the dependency specified under 
python.pip_packages
This internally uses pip dependency resolver, this means that it has one issue:
The locked version in this bento will use the latest library version instead of the current version in your local environment
If you take a look, the locked xgboost version is actually 1.6.2 instead of the one you use to train (could be a different version) The issue that you encounter is that the model is trained in a different xgboost comparing to the xgboost built inside the container. One fix that you can do right now is to lock the xgboost version that is used to train your model into the bentofile.yaml
Copy code
python:
  pip_packages:
  - xgboost==1.5.0 # or the version of xgboost in your system
This will ensure that the model inside the container will use the correct xgboost version
We are going to add an environment manager to hope address this issue like this. I will ping you once we have this feature in
i
Thank you! instead of locking the version in the bentofile.yaml I ran all the cells in the notebook over again to retrain the model (as mentioned above I upgraded xgboost to 1.6.2). It appears to be working now.
I also got a warning about the dictvectorizer coming from a different version of scikit-learn so I set the version in the yaml file. How often should i be updating these packages.? I just installed that version of scikit-learn yesterday when I reinstalled anaconda.
3 Views
Open in Slack
PreviousNext
Powered by