This message was deleted.

Try

--enable-microbatch

option. I'm not sure if this is the problem, but can be.

Hi @João Victor Davim, thank you! This solved my problem. However, the docs do mention that micro-batching is enabled by default since 0.12.0 and I'm on 0.12.1 so I'm not sure why this is happening.

Actually i'm not a collaborator yet. But maybe @Chaoyu would help you with this. Anyway, I think you can open an issue at https://github.com/bentoml/BentoML/issues to change this behavior.