I've recently implemented batching for a couple of our models and am noticing some odd results. Namely, the server can handle much smaller loads even when I try to force it to use batch sizes of 1, as if batching weren't enabled (right now I'm just trying to iteratively build up to larger batch sizes. Details in 🧵

what is the latency of the runner's prediction? Our scheduler for the batching mechanism will try to respect this setting

{"max_batch_size": 2, "max_latency_ms": 100}

The error you've seen usually means the scheduler is rejecting additional request because its unable to satisfy the latency requirement.

The latency of the runner is probably right around there, but we have a second runner I'm testing that has a latency that is much larger.

batch size 1 = no batching on the implementation level

and we will review this! thanks for spotting

Thanks for the help here! Follow up: do y'all have documentation on best practices for the

max_latency_ms

parameter? I've got a runner with latency around ~1s, and I'm seeing that with load testing it's best to have

max_latency_ms

around 32000 (32 seconds) for batching to maximize throughput. But, in production we can't actually afford to have requests take an extra 32 seconds, so I'm wary of actually using that 32000 value.

you should set

max_latency_ms

accordingly to the SLA of your service. in practice to maximize your throughput and latency, it will be a combination of horizontal + vertical scaling,+ adaptive batching which is what bentoml offers. if you cannot afford to have a high latency, i would suggest having an acceptable

max_latency_ms

and scale your services horizontally, either using more workers / have multiple replicas

@Jian Shen Yap supposing we didn't have an SLA, what would be the recommended optimal setting for

max_latency_ms

? Would it be some multiple of the latency of the runner?

recommended optimal setting for

max_latency_ms

There's no single answer for this. "Optimal" depends on what are you trying to optimize for. You will have to play around with different combinations of settings and see what fits you the best!

@Jian Shen Yap if we want to optimize for throughput, are there recommendations of where to start, e.g. take the latency of the runner and multiply it by 2 or something? Anything like that? I'm not seeing anything that precise in the docs for batching, but apologies if I've missed it!

reject the additional requests

it will predict if it can satisfy

max_latency_ms

, if not, it will reject the request.

latency of the runner and multiply it by 2

i think you can start by multiple of 2/3. But this is also depends on how batching is implemented for your model. We can assume most model implement batching correctly, then the above setting is a good starting point. It also depends on the resource utilization, having batching means it will take up more compute resource on your CPU/GPU.

This is all very helpful, thanks @Jian Shen Yap! A few follow ups:

it will predict if it can satisfy

max_latency_ms

, if not, it will reject the request

Gotcha, this is very helpful to know. When I'm performing load testing with batching, it seems that I'm getting 503 errors (server busy) more quickly than without batching. I'm suspecting that it's because of the reject mechanism here, where with batching, the requests get rejected because of this

predict if can satisfy max_latency_ms

condition, whereas without batching they would just timeout according to the runner timeout. Is that right? Or how does the non-batching case work in terms of rejecting requests to a runner?

But this is also depends on how batching is implemented for your model. We can assume most model implement batching correctly, then the above setting is a good starting point.

Out of curiosity, what are the wrong / right ways to have batching implemented? We're not seeing as large of speed gains from batching as we had thought, so perhaps we do have it implemented incorrectly.

more quickly than without batching. I'm suspecting that it's because of the reject mechanism here, where with batching, the requests get rejected because of this

predict if can satisfy max_latency_ms

condition, whereas without batching they would just timeout according to the runner timeout. Is that right? Or how does the non-batching case work in terms of rejecting requests to a runner?

You are right! if you relax the

max_latency_ms

you should see that less. Generally theres two scenario that you will see this error: 1. the http Server is overloaded and is not responsive. 2. the scheduler rejects the request ( only if batching it turned on )

You are right! if you relax the

max_latency_ms

you should see that less.

Okay rad, I think I'm starting to grok this. One more question to check my understanding. In effect, when we turn on batching, the

max_latency_ms

is acting as the runner timeout value, yeah? Gotcha on the batching! We're not implementing batching in that way 😅 .

One way to measure how batching is properly implemented, is predict 10 request at once takes almost the same time as predicting 1 request. Usually most DL model is built to support batch predict.

Out of curiosity, and this is now down the DL rabbit hole a bit, but do you have any references that show inference time speedups with batching? I haven't actually been able to find much via google - most references I've seen mention training time. Here's a chart of the speedup we've been seeing across different models (model names blurred for privacy), and it's not as high as I would have thought, and nowhere near the "predicting on X requests is nearly the same as 1 request" (which is roughly what we thought we'd see when doing batching). And the pseudocode way we've been doing batching is basically:

dl = PyTorchModel()

input = TorchTensor()
for batch_size in [1, 2, 4, 8, 16, 32]:
    batched_input = torch.stack([input] * batch_size)
    dl(batched_input)

also you can turn on

bentoml serve --debug

there you can see some of the "Scheduler" logs and the dynamic batch sizes

also you can turn on

bentoml serve --debug

there you can see some of the "Scheduler" logs and the dynamic batch sizes

Ahhh, amazing thank you!

Hello Sean, i realised i left you hanging here! my bad! Here's a visualization of what happens the dispatch will try to respect the condition

max_latency_ms <= time_waiting_in_batch + request_processing_time

@Jian Shen Yap all good! I know I've been hitting ya with a lot of questions and there are lots of other folks on here with questions - appreciate you circling back! And I appreciate all the info you're bringing here - we've got Bento deployed in production but there's a lot of stuff in the weeds here that we don't understand, and I think understanding it is crucial to really getting it to work at scale for us (we've recently scaled up quite a bit). This info has been really helpful! Something I've noticed in recent testing... If I leave the

max_batch_size

as say, 4, and the

max_latency_ms

unspecified (which puts it at the default), and then perform a load test with 1 user (so requests coming in fairly slowly, but still at least 1 per second) - the runner scheduler doesn't wait the full 60000 ms for a batch. It instead appears to converge to a batch size of 1 and just constantly take a batch of 1. It seems that the scheduler is taking the

max_batch_size

and

max_latency_ms

as upper bounds, and then using some other algorithm to decide how long it will actually wait for another request. Do y'all have more detail on exactly how that algorithm works?