This message was deleted.

Successful request:

preprocessing image!
run inference
Handling batch of size: (1, 224, 224, 3)
2022-12-07T08:33:31+0000 [INFO] [api_server:3] 10.0.151.110:35104 (scheme=http,method=GET,path=/healthz,type=,length=) (status=200,type=text/plain; charset=utf-8,length=1) 0.480ms (trace=8c38ddb97443eb11923895910c1a4537,span=8842f24da5a218e2,sampled=0)

1/4 [======>.......................] - ETA: 0s
2/4 [==============>...............] - ETA: 0s
3/4 [=====================>........] - ETA: 0s
4/4 [==============================] - ETA: 0s
4/4 [==============================] - 1s 170ms/step
(1, 224, 224, 3)
2022-12-07T08:33:32+0000 [INFO] [runner:generateheatmaprunnable:1] _ (scheme=http,method=POST,path=/generate_image_overlayed_with_heatmap,type=application/octet-stream,length=1204633) (status=200,type=application/vnd.bentoml.NdarrayContainer,length=150694) 1246.341ms (trace=b4ee700f11927e849a76aeac524e82ce,span=40c40a502e1915ac,sampled=0)

This was immediately followed by this failed request in the logs:

2022-12-07T08:33:32+0000 [ERROR] [api_server:1] Exception on /predict-image [POST] (trace=00fd7769466390c53b99e314a1d65188,span=65462bfd10cf3487,sampled=0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/server/http_app.py", line 338, in api_func
    response = await api.output.to_http_response(output, ctx)
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/io_descriptors/multipart.py", line 275, in to_http_response
    resps = await asyncio.gather(
  File "/usr/local/lib/python3.8/dist-packages/bentoml/_internal/io_descriptors/image.py", line 351, in to_http_response
    image = PIL.Image.fromarray(obj, mode=self._pilmode)
  File "/usr/local/lib/python3.8/dist-packages/PIL/Image.py", line 3004, in fromarray
    raise ValueError(f"Too many dimensions: {ndim} > {ndmax}.")
ValueError: Too many dimensions: 4 > 3.
2022-12-07T08:33:32+0000 [INFO] [api_server:1] 10.0.151.110:35088 (scheme=http,method=POST,path=/predict-image,type=multipart/form-data; boundary=2b8f84fea702ef2bb808784b708dd6c6,length=1146305) (status=500,type=application/json,length=110) 1300.823ms (trace=00fd7769466390c53b99e314a1d65188,span=65462bfd10cf3487,sampled=0)

Hmmm is doing batching or something

My locust file is fairly trivial so I'll include it here ^^^

Not the locust file but the actual Bento service

Sorry, was in the middle of typing that out when you messaged.

Is it doing batching or something?

Yes it is. And my co-worker and I did observe this problem locally when we initially turned on batching. But then it mysteriously went away again. We were able to watch as a few batches both of size 1 and of size 3 were processed okay.

huh that indeed is weird

Good question. I'll run it locally real quick and test

Yeah I noticed while looking at the stack trace

Just ran it locally with out the

--production

flag and saw that it's doing this over and over again:

preprocessing image!
run inference
Handling batch of size: (1, 224, 224, 3)
4/4 [==============================] - 1s 181ms/step
(1, 224, 224, 3)
2022-12-07T02:00:58-0700 [ERROR] [dev_api_server] Exception on /predict-image [POST] (trace=06a1f498a82082ba94116d220126539f,span=4fb2d8a53385a7a2,sampled=0)
Traceback (most recent call last):
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/_internal/server/http_app.py", line 338, in api_func
    response = await api.output.to_http_response(output, ctx)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/_internal/io_descriptors/multipart.py", line 279, in to_http_response
    resps = await asyncio.gather(
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/_internal/io_descriptors/image.py", line 351, in to_http_response
    image = PIL.Image.fromarray(obj, mode=self._pilmode)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/PIL/Image.py", line 3004, in fromarray
    raise ValueError(f"Too many dimensions: {ndim} > {ndmax}.")
ValueError: Too many dimensions: 4 > 3.
2022-12-07T02:00:58-0700 [INFO] [dev_api_server] 127.0.0.1:56117 (scheme=http,method=POST,path=/predict-image,type=multipart/form-data; boundary=2cc686ade7961c3c98592d976efb935b,length=1146305) (status=500,type=application/json,length=110) 1273.850ms (trace=06a1f498a82082ba94116d220126539f,span=4fb2d8a53385a7a2,sampled=0)

this is with or without batching?

With. Running it run again without, but wanted to see if it was only occurring during batching when the batch size is greater than 1

👍

And I'm wondering: it seems that the model is handling batches okay and THEN it's failing. Could it be that my shape is bad when I try to return it to the user?

Ohhh

Printing the shape...

yeah if your model can perform inference then that's actually good

Well would you look at that:

shape of image about to be returned: (1, 224, 224, 3)

hmmmm

Yaaaay! Success!

Woo hoo!

I added this if statement

if len(image_with_heatmap.shape) == 4:
        image_with_heatmap = image_with_heatmap[0]

lol

I'm glad you thought about the sqeeze/batch_dim thing, too. I'll have to double check we're doing that. Thanks for your help!

Happy to help