This message was deleted BentoML #ask-for-help

Join Slack

This message was deleted.

# ask-for-help

Slackbot

10/20/2022, 7:25 PM

This message was deleted.

Sudeep Ghimire

10/21/2022, 1:21 AM

@Jiang any idea on 👆 scenario?

Jiang

10/21/2022, 2:28 AM

I guess you saved the

demo_img_loader

as a Pickleable model, correct?

Jiang

10/21/2022, 2:36 AM

I have two advices. 1. The reason a runner works under dev server but doesn't under prod server is typically it is not saved properly. It's very common in Pickleable and PyTorch models, since both of them are using pickle to save/restore models. See the official doc here: https://docs.bentoml.org/en/latest/frameworks/pytorch.html#id4 to make it work. 2. For another level, it is not recommended to do IO ops in runners. If

demo_img_loader

is a bare Python async function, you can await call it directly in the endpoint (the

predict

function). If it is a sync function, you may make your endpoint a sync function and call it directly.

Sudeep Ghimire

10/21/2022, 10:08 AM

@Jiang thnx for the guide, totally agree with #1 above, and went via #2 and continued getting the same error when running server with --production flag.

Copy code

@router.post("/analyze")
async def predict(inference_request: InferenceRequest):
    pil_img: Image = await async_load_image_internal(inference_request.s3Path)
    print('-------Main----')
    print(pil_img.is_loaded_by_image_loader)
    print(pil_img.loading_status)
    print('-------')
    results = await asyncio.gather(
        demo_runner1.infer.async_run(pil_image), demo_runner2.infer.async_run(pil_image),
        demo_runner3.infer.async_run(pil_image)
    )
    return results

image loaded by async_load_image_internal has correct attributes for

is_loaded_by_image_loader

and

loading_status

when checking in the logic for the service endpoint, but check the same in the infer methods for the runners fails. Really getting out of ideas to resolve the issue. Is it because of the case that endpoint is defined

@router.post("/analyze")

i.e. decorator for router from fastapi ( instead of using the decorator

@svc.api

) and registered in service.py as

Copy code

fastapi_app = FastAPI()
fastapi_app.include_router(inference_router)
svc = bentoml.Service("demo_abc",
                      runners=[demo_runner1, demo_runner2, demo_runner3])
svc.mount_asgi_app(fastapi_app)

Sudeep Ghimire

10/21/2022, 10:20 AM

Most interestingly to the above case, if i load the pil_image in the

infer

method of individual runners then it works as expected. But loading initially in the part of the endpoint defn and passing pil image to the

infer

method of runners is not working

Jiang

10/21/2022, 5:44 PM

ah. I don't think the original PIL.Image has a

is_loaded_by_image_loader

attribute. So the class of

pil_img

here is a custom class?

Jiang

10/21/2022, 5:50 PM

This might be due to the limitation of runners. Could you share the error stack here?

2 Views

Open in Slack

Previous Next