Hi guys, I updated my bento to implement some logging. To do so I had to upgrade my bentoml version from

1.3.0

to

1.3.5

and i'm facing some issues. Here is the previous version of my code (on bentoml

1.3.0

)

class SpatialTokenClassifierInput(BaseModel):
    """Pydantic class for the inputs of the inference API method of the SpatialTokenClassifier"""

    sample: str | bytes = Field(description="Path to the file (image or pdf) or byte stream.")
    page_indexes: list | None = Field(
        description="A numpy array containing the index of each page to extract. If None extracts every pages."
    )

@bentoml.service(
    name='spatial-token-classifier',
    resources={"cpu": "cpu_count"},  # Adjust as needed
)
class SpatialTokenClassifierService:
    """The Runner takes care of running the inference of the Yolo model.

    Attributes:
        model (LiltModel): Instance of the LiltModel for inference.
        inference_size (int): Size for inference.
    """

    def __init__(self) -> None:
        """Loads the model and tokenizer."""
        # Load the model
        self.model = load_spatial_token_classifier()

        # Config inference settings
        self.inference_size = 320

    @bentoml.api(input_spec=SpatialTokenClassifierInput)
    def inference(self, sample: str | bytes, page_indexes: list | None = None) -> Any:
        return self.model.inference(sample, page_indexes=page_indexes)

    @bentoml.on_deployment
    def download_model_on_serve() -> None:
        load_spatial_token_classifier()

I was querying my bento like this :

headers = {"Authorization": f"Bearer {id_token}"}
doc_bytes = open(path, "rb")
files = {
    'file': (path, doc_bytes, 'application/pdf'),
    'page_indexes': str(index_pages),
}
response = requests.post(url, headers=headers, files=files, timeout=300)

Here is the new version of my code (on bentoml

1.3.5

)

class SpatialTokenClassifierInput(IODescriptor):
    """Pydantic class for the inputs of the inference API method of the SpatialTokenClassifier"""

    sample: str | bytes = Field(description="Path to the file (image or pdf) or byte stream.")
    page_indexes: List[int] | None = Field(default=None,
        description="A numpy array containing the index of each page to extract. If None extracts every pages."
    )


@bentoml.service(
    name='spatial-token-classifier',
    resources={"cpu": "cpu_count"},  # Adjust as needed
    traffic={"timeout": 600},
    logging={
        "access": {
            "enabled": True,
            "request_content_length": True,
            "request_content_type": True,
            "response_content_length": True,
            "response_content_type": True,
            "skip_paths": ["/metrics", "/healthz", "/livez", "/readyz"],
            "format": {"trace_id": "032x", "span_id": "016x"},
        }
    },
)
class SpatialTokenClassifierService:
    """The Runner takes care of running the inference of the Yolo model.

    Attributes:
        model (LiltModel): Instance of the LiltModel for inference.
        inference_size (int): Size for inference.
    """

    def __init__(self) -> None:
        """Loads the model and tokenizer."""
        # Load the model
        self.model = load_spatial_token_classifier()

        # Config inference settings
        self.inference_size = 320

        # Set up logging
        console_handler = logging.StreamHandler()
        formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
        console_handler.setFormatter(formatter)
        self.logger = logging.getLogger("bentoml")
        self.logger.addHandler(console_handler)
        self.logger.setLevel(logging.INFO)

    @bentoml.api(input_spec=SpatialTokenClassifierInput)
    def inference(self, sample: str | bytes, page_indexes: list | None = None) -> Any:
        """Runs inference on provided sample on given pages"""
        self.logger.info('Inference called')
        self.logger.info('Sample : %s', sample)
        self.logger.info('Page indexes : %s', page_indexes)
        response_dict = self.model.inference(sample, page_indexes=page_indexes)
        for key, value in response_dict.items():
            self.logger.info("%s: %s", key, value)

        return response_dict

    @bentoml.on_deployment
    def download_model_on_serve() -> None:
        load_spatial_token_classifier()

As you can see I didn't change much, I simply added some logging-related code (and upgraded bentoml version) But when I run the same python query I get the following error:

[ERROR] [entry_service:spatial-token-classifier:1] Exception on /inference [POST] (trace=d9ce8f11d98b6be73f6f7491c7433bbe,span=31c2c6c5a0c8b07e,sampled=0,service.name=spatial-token-classifier)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 562, in api_endpoint_wrapper
    resp = await self.api_endpoint(name, request)
  File "/usr/local/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 622, in api_endpoint
    input_data = await method.input_spec.from_http_request(request, serde)
  File "/usr/local/lib/python3.10/site-packages/_bentoml_sdk/io_models.py", line 171, in from_http_request
    return await serde.parse_request(request, t.cast(t.Type[IODescriptor], cls))
  File "/usr/local/lib/python3.10/site-packages/_bentoml_impl/serde.py", line 200, in parse_request
    assert isinstance(v := form[k], str)
AssertionError

I couldn't find a way to make the python query work, do you have any ideas ? 🙏

I think most commonly we see the use of Pydantic model like this:

@bentoml.api
    def inference(self, data: SpatialTokenClassifierInput) -> Any:

I’m not sure how the

input_spec

would map to function params. @Frost Ming any insights here?

It seems like to me you don’t need any special serde here, would the following be suffice?

@bentoml.api
def inference(self, sample: str | bytes, page_indexes: list | None = None) -> Any: ...

BentoML will automatically convert all kwargs into a Pydantic Model

Thanks for your answers ! I moved

load_spatial_token_classifier

to

__init__

as suggested About the api definition I tried both your suggestion and I still get the same isinstance AssertionError. I don't know if the problem comes from the api definition or the way the pdf is send with the

<http://requests.post|requests.post>()

is the

sample

suppose to take a PDF file?

Thank you @Chaoyu Here is the new version of my api definition

@bentoml.api()
def inference(
        self,
        pdf: Annotated[Path, bentoml.validators.ContentType("application/pdf")],
        page_indexes: Optional[list] = None,
) -> Any:
    <http://self.logger.info|self.logger.info>('Inference called')
    <http://self.logger.info|self.logger.info>('pdf : %s', pdf)
    <http://self.logger.info|self.logger.info>('page_indexes : %s', page_indexes)
    response_dict = self.model.inference(pdf, page_indexes=page_indexes)
    for key, value in response_dict.items():
        <http://self.logger.info|self.logger.info>("Page %s: %s", key, value)

    return response_dict

This definition seems to works, when using this curl command the api works as expected :

curl -X 'POST'   '<http://localhost:3000/inference>'
    -H 'accept: application/pdf'
    -F pdf=@/some/path/to/local/file.pdf
    -F page_indexes='[0]'

But when I run the python query :

files = {
    'pdf': pdf_path,
}
response = <http://requests.post|requests.post>(full_url, headers=headers, files=files, timeout=240)

The file seems to be copied on a tmp folder on the bento side, which is a good thing (the logs say that pdf = '/tmp/bentoml-request-8arhbz7l/tmp54_t47gqpdf') But when I try to load the pdf it says

cannot open broken document

so i'm still struggling at finding a working python query

Your Python code with

requests.post

and your CURL command are not equivalent, please refer to

requests

library’s documentation

I was trying to find a way to not use the BentoML client to avoid installing BentoML on my project calling the API 😅 It's working perfectly with the client so i'll go for it, thanks a lot for your help 🙏