however these are resolved with the workaround as per the issue

when micro batching is enable, i get such error.

[2020-11-13 17:10:08,117] ERROR - Traceback (most recent call last):
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 210, in request_dispatcher
    resp = await self.batch_handlers[api_name](req)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/dispatcher.py", line 141, in _func
    raise r
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/dispatcher.py", line 201, in outbound_call
    outputs = await self.callback(tuple(d for _, d, _ in inputs_info))
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 109, in _batch_handler_template
    return await func(requests, api_name)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 286, in _batch_handler_template
    return tuple(
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/bentoml/marshal/marshal.py", line 287, in <genexpr>
    aiohttp.web.Response(body=i.body, headers=i.headers, status=i.status)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/aiohttp/web_response.py", line 540, in __init__
    super().__init__(status=status, reason=reason, headers=real_headers)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/aiohttp/web_response.py", line 86, in __init__
    self.set_status(status, reason)
  File "/Users/jianshen/Desktop/Projects/reco-engine-v2/venv/lib/python3.8/site-packages/aiohttp/web_response.py", line 119, in set_status
    self._status = int(status)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

i printed out

i.status

-> it is

None

. For testing purposes I modify is so that it will return 200 when

i.status

is None , and it successfully ran with the right prediction

What would be the possible cause for the api server to return status code of None when using microbatching?

In normal mode without micro batched enable, when i passed a dataframe with two rows [ […] , […]], I would get a prediction with 2 rows as well.

For extra information of the model This is the model summary of our model. It is built using keras functional api. We had to specify a batch size because we are using stateful GRU. When loaded as TensorflowSavedModel, the batch size shape will be

None

.

Let me know if any extra information is needed!

I believe bentoml should handle this exception and provide better feedback when user returned mismatched output. I’ll open an issue to track this.

class KerasModelService(bentoml.BentoService):
    @bentoml.api(input=DataframeInput(), mb_max_latency=1000, batch=True)
    def predict(self, df: pd.DataFrame):
        int_oh = np.append(np.zeros((2048 - df.shape[0], df.shape[1])),df,axis=0)
        exp_int_oh = np.expand_dims(int_oh, axis=1)
        batch_preds = self.artifacts.model(exp_int_oh.astype('float32'))
        preds_idx = tf.argsort(batch_preds,axis=-1,direction='DESCENDING',stable=False,name=None)
        rev = tf.reverse(preds_idx,[0])
        return "testing"

When i tested with a dummy string above, it will automatically truncate the string “testing” according to the size of the batch input, and it doesn’t throw an error for

i.status

, which makes me think that it is not the output length mismatch that causes this error to occur

After more debugging, i found out that

rev

is type of

EagerTensor

. After converting it to a

ndarray

and return it, microbatching works!

I am testing with a single request with the bentoml server running at microbatching mode. Perhaps there might be some incompatibility with tensor outputs?