https://bentoml.com logo
Join Slack
Powered by
*Error occured when define bentoml service using ...
# ask-for-help
y
Error occured when define bentoml service using 
class
interface format
✅ 1
Hi, I am newbie in BentoML. I am trying to deploy BentoML in local server. Trained model on BentoML is trained by MLflow library, and I import it to BentoML Model Store. I defined BentoML using functional interface format of api below format and I succeed to deploy it.
Copy code
import bentoml
import os

from <http://bentoml.io|bentoml.io> import NumpyNdarray
from numpy import ndarray


bento_svc_name = os.getenv("BENTOML_SVC_NAME")
bento_model_name = os.getenv("BENTOML_MODEL_NAME")
bento_model = bentoml.mlflow.get(f"{bento_model_name}:latest")
bento_runner = bento_model.to_runner()

svc = bentoml.Service(
    name=bento_svc_name,
    runners=[bento_runner]
)


def preprocess(x: ndarray) -> ndarray:
    return x 


@svc.api(input=NumpyNdarray(), output=NumpyNdarray())
def predict(x: ndarray) -> ndarray:
    x = preprocess(x)
    y = bento_runner.predict.run(x)
    return y
But, I defined the same server using class interface format of api. this code is below.
Copy code
import bentoml
import os

from pydantic import BaseModel, Field


bento_svc_name = os.getenv("BENTOML_SVC_NAME", "tabular-diabetes-regressor-v1-bento-svc")
bento_model_name = os.getenv("BENTOML_MODEL_NAME", "tabular-diabetes-regressor-v1-bento-model")
bento_model = bentoml.mlflow.get(f"{bento_model_name}:latest")
bento_runner = bento_model.to_runner()


class InputRequest(BaseModel):
    instances: list[float] = Field(example=[0,1,2,3,4,5,6,7,8,9])


class OutputResponse(BaseModel):
    y: float


@bentoml.service(runners=[bento_runner])
class ZeddTest:
    def __init__(self) -> None:
        self.bento_runner = bento_runner

    @bentoml.api(input_spec=InputRequest, output_spec=OutputResponse)
    def predict(self, instances):
        y = self.bento_runner.predict.run(instances)
        return y
And I command 
bentoml serve service:ZeddTest
and when on swagger-ui I request to server and encountered this error.
Copy code
Traceback (most recent call last):
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 418, in api_endpoint_wrapper
    resp = await self.api_endpoint(name, request)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 508, in api_endpoint
    output = await self._to_thread(func, *input_args, **input_params)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 363, in _to_thread
    output = await anyio.to_thread.run_sync(func, limiter=self._limiter)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
    return await future
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run
    result = context.run(func, *args)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_sdk/method.py", line 131, in wrapped
    return self.func(instance, *args, **kwargs)
  File "/Users/zedd.ai/home-mlops/serving/tabular-diabetes-regressor/bento-ml/service.py", line 28, in predict
    y = self.bento_runner.predict.run(instances)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 53, in run
    return self.runner._runner_handle.run_method(self, *args, **kwargs)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/bentoml/_internal/runner/runner_handle/__init__.py", line 71, in run_method
    raise StateException(
bentoml.exceptions.StateException: Runner is not initialized. Make sure to include '<bentoml._internal.runner.runner_handle.DummyRunnerHandle object at 0x16e256440>' to your service definition.
What is the reason of this error? I asked ChatGPT and googling but I can't fix it.. plz advise me!
j
Hello, BentoML service is a 1.2 version API, which removes the concepts of runner. instead of 
to_runner()
you will have to do a 
runner_service()
conversion instead. https://docs.bentoml.com/en/latest/guides/services.html#convert-legacy-runners-to-a-service If you want to use the runner method, you will have to use the legacy API, 1.1 where you can see the documentation here. https://docs.bentoml.com/en/v1.1.11/concepts/service.html
y
@Jian Shen Yap Thanks for your reply. I got it. More, I have another question. According to your attached docs, starting whith BentoML 1.2+, 
runner
component is no longer supported and using 
runner_service()
is only for migration for legacy code. So, I wonder the official code when making BentoML with model trained by MLflow. Is guideline exist?
@Jian Shen Yap Hi. I follw your suggestion based on attached docs and anyway I succeed in fixing the error. but I found the wrong guideline in the docs. According to docs, the example code is below with my case using 
diabetes toy-dataset
• Based on docs
Copy code
import bentoml
import os

from numpy import ndarray


bento_svc_name = os.getenv("BENTOML_SVC_NAME", "tabular-diabetes-regressor-v1-bento-svc")
bento_model_name = os.getenv("BENTOML_MODEL_NAME", "tabular-diabetes-regressor-v1-bento-model")
bento_model = bentoml.mlflow.get(f"{bento_model_name}:latest")
bento_legacy_runner = bento_model.to_runner()
bento_runner_svc = bentoml.runner_service(runner=bento_legacy_runner)


@bentoml.service(
    traffic={"timeout": 10},
    resources={"cpu": "2"},
    name=bento_svc_name
)
class DiabetesRegressor:
    runner = bentoml.depends(bento_runner_svc)
    
    @bentoml.api
    def predict(self, x: ndarray) -> ndarray:
        x = self.preprocess(x)
        # print(type(self.runner.predict), dir(self.runner.predict))
        y = self.runner.predict.run(x)
        return y
    
    def preprocess(self, x: ndarray) -> ndarray:
        return x
and run command 
bentoml serve service:DiabetesRegressor
and call 
/predict
API, below error occured.
Copy code
Traceback (most recent call last):
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 418, in api_endpoint_wrapper
    resp = await self.api_endpoint(name, request)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 508, in api_endpoint
    output = await self._to_thread(func, *input_args, **input_params)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 363, in _to_thread
    output = await anyio.to_thread.run_sync(func, limiter=self._limiter)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
    return await future
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run
    result = context.run(func, *args)
  File "/Users/zedd.ai/.pyenv/versions/home-mlops/lib/python3.10/site-packages/_bentoml_sdk/method.py", line 131, in wrapped
    return self.func(instance, *args, **kwargs)
  File "/Users/zedd.ai/home-mlops/serving/tabular-diabetes-regressor/bento-ml/service.py", line 26, in predict
    y = self.runner.predict.run(x)
AttributeError: 'function' object has no attribute 'run'
So, I verbose type and dir of 
self.runner.predict
and I check it is just function and has a callable method (
___call___
) So, I changed my code removing 
run
• Changed code
Copy code
import bentoml
import os

from numpy import ndarray


bento_svc_name = os.getenv("BENTOML_SVC_NAME", "tabular-diabetes-regressor-v1-bento-svc")
bento_model_name = os.getenv("BENTOML_MODEL_NAME", "tabular-diabetes-regressor-v1-bento-model")
bento_model = bentoml.mlflow.get(f"{bento_model_name}:latest")
bento_legacy_runner = bento_model.to_runner()
bento_runner_svc = bentoml.runner_service(runner=bento_legacy_runner)


@bentoml.service(
    traffic={"timeout": 10},
    resources={"cpu": "2"},
    name=bento_svc_name
)
class DiabetesRegressor:
    runner = bentoml.depends(bento_runner_svc)
    
    @bentoml.api
    def predict(self, x: ndarray) -> ndarray:
        x = self.preprocess(x)
        print(type(self.runner.predict), dir(self.runner.predict))
        y = self.runner.predict(x)   # it is changed!
        return y
    
    def preprocess(self, x: ndarray) -> ndarray:
        return x
and run 
bentoml serve service:DiabetesRegressor
and call 
/predict
API, I got a 200 status. please modify content of docs for anyone who has trouble with this problem. Thanks!
Open in Slack
PreviousNext
Powered by