This message was deleted.

BentoML does server-side dynamic batching - so in the case where client side batching is not available, BentoML has you covered. And it automatically adjust wait time and batch size for optimal performance 😄