This message was deleted.

yes you will have to use multipart, but we are releasing a new version of bentoml which will natively support

list[Image]

next month!

Great to know! Then, I'll come up with a single Image first wait for your updates towards list[Image] However, would you mind to give an example on how to apply Multipart for multiple images.

I'm trying to do the same now, do you have any pointers please @Sergej Dogadov @Jian Shen Yap? I've checked the latest bentoml releases and can't see support for

list[Image]

yet

i believe this is possible

list[Image]

with the 1.2 SDK!

import bentoml
from PIL.Image import Image

@bentoml.service
class MnistPredictor:
    @bentoml.api
    def infer(self, image: list[Image])-> int:
        return 1

Oh so you wouldn't use the bentoml.io.Image descriptor:

@svc.api(input=list[bentoml.io.Image()], output=bentoml.io.JSON(), route="/image_ocr")
async def ocr(file) -> dict:
    encoded_images = await read_file(file)
    result = await ocr_pipeline.recognise_text([encoded_images])
    return result[0]

as this didn't work for me?

oh, list of X doesn't work for bentoml <1.2!

On

bentoml==1.2.4

for this:

@svc.api(
    input=list[bentoml.io.Image()], output=bentoml.io.JSON(), route="/image_ocr/batch"
)
async def ocr_batch(files) -> dict:
    encoded_images = await asyncio.gather(*[read_file(file) for file in files])
    results = await ocr_pipeline.recognise_text(encoded_images=encoded_images)
    return results

I get:

_api = InferenceAPI[IOType](
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.pyenv/versions/3.11.7/lib/python3.11/typing.py", line 1275, in __call__
    result = self.__origin__(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ocr/venv/lib/python3.11/site-packages/bentoml/_internal/service/inference_api.py", line 50, in __init__
    input_type = input_descriptor.input_type()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'list' has no attribute 'input_type'

Is this not expected?

you would not use the IO descriptor anymore with the new SDK. https://docs.bentoml.com/en/latest/guides/iotypes.html service definition is also revamped! https://docs.bentoml.com/en/latest/guides/services.html

Ahh i see - sorry i missed the revamp. Thanks @Jian Shen Yap!

we should have probably be more clear with our docs! we will work on that

I'd just been using bentoml a lot the last few months to deploy Whisper and had learnt how to use bentoml, so wasn't expecting a new interface in the same major version! Looking forward to trying it though. Interesting that configuration is no longer controlled by the yamls now too!

Yaml are still available! but doing it in code we felt its closer to the right mental model

Makes sense 👍