BentoML

image.png

Hi Steven, did you save the model with `batchable=True` like here: <https://docs.bentoml.org/en/latest/frameworks/transformers.html#adaptive-batching> ?

<@U023HR5862J> Yes I did. Do I need to remove it?

```result = await runner.async run([inputs .dict ()])```

then change your `preprocess` function to handle a list of dict

I see. Let me try. But I also wanna ask if I set it to True, does that mean I need to handle the batch inference in the predict function?

Yep. If your model is more efficient when doing batch inference, then I suggest you try to find a way to make it work. Else you may just turn off batching

Cool. I just tried. And it act as I expect now. Thanks so much! <@U023HR5862J> 