Hi there 👋 This afternoon I have been banging my head against the wall with the Adaptive Batching feature in BentoML 1.2.x, there's something I must have missed because I can't seem to be able to make it work. The way I understand the feature is: • It is possible to declare a service HTTP endpoint as "batchable" • From an HTTP request standpoint, the endpoint would accept a typical payload, let's say a

MyClass

entity such that:

{"a": "my string"}

• From a BentoML service method declaration standpoint, the endpoint is expected to accept an iterable argument, so a

typing.List[MyClass]

• Within the method itself, the iterable argument can be manipulated (eg. submitted to a model), and the method's return value should be an iterable which order matches that of the argument • BentoML handles all of the batching part, will call the method with as many

MyClass

entities in the iterable as could be batched given the batch_size, latency settings, etc., and will dispatch individual responses to the endpoint callers This is how I understand it conceptually, but in practice all I manage to create is an endpoint that works indistinguishably from a non-batchable endpoint. So what's the catch?

From a user perspective, you will now pass a list of items. within bentoml, we will concatenate these items if it falls within the batching window and process them together. If you perform a load test, and your processing logic benefits from batching, you should see an increase in throughput as compared to the non batching version, especially during high load siutation

Is it possible for adaptive batching to not trigger even if concurrent calls should fall within batching parameters? (i.e. say 3 HTTP calls simultaneously pushing 3 items each, for a

max_batch_size

of 25) Are there some specific settings to check (like concurrency factors? something else?) that could render adaptive batching inoperant?

We don't support such control yet for our OSS