This message was deleted.

if you could tolerate higher latency from the runner, maybe you can try increasing

max_latency_ms

? you could also run

bentoml serve --debug

to see more logs related to the adaptive batching if you think the its the batching algorithm is not doing the right thing!