Hi there folks, I have been trying to make Adaptive Batching work for Pydantic style input formats and have been following this tutorial https://docs.bentoml.com/en/latest/guides/adaptive-batching.html#handle-multiple-parameters But when I run the Locust test locally for example to mimic multiple requests withing batch time frame, the server doesn't batch at all, it processes request by request. I have tried multiple values for batching parameters, non seem to make it work. Anyone have an idea why? Would really appreciate the help as we are trying to use BentoML in our company. 🙂 Thanks! Here is the example server code:

@bentoml.service(
    name="SignatureExtractor",
    metrics={"enabled": False},
    logging={
        "access": {
            "enabled": True,
            "skip_paths": ["/metrics", "/healthz", "/livez", "/readyz"],
        }
    },
)
class SignatureExtractor:
    def __init__(self) -> None:
        self.model: dict = {}  # TODO: define model loading
        self.logger: any = StructLogger().logger

    @bentoml.api(batchable=True, max_batch_size=32, max_latency_ms=5000)
    def batch_extract(self, inputs: list[ExtractorInput]) -> list[dict]:
        <http://self.logger.info|self.logger.info>("Input message", body=inputs, length=len(inputs))
        response: list[dict] = [{"output": input} for input in inputs]
        return response


@bentoml.service(
    name="WrapperSignatureExtractor",
    metrics={"enabled": False},
    logging={"access": {"enabled": False}},
)
class WrapperSignatureExtractor:
    batch = bentoml.depends(SignatureExtractor)

    @bentoml.api(route=f"{parameters.API_ROOT_PATH}/extract")
    def wrap(self, id: str, description: str) -> dict:
        result: list[dict] = self.batch.batch_extract(
            inputs=[ExtractorInput(id=id, description=description)]
        )
        return result[0]

I start the server with this command:

bentoml serve service:WrapperSignatureExtractor

Hello, i had a similar issue, which i solved using async routes in both services.

@Lounès Lounis Thanks so much, you are awesome, that actually solved it, now it batches requests!

@Frost Ming if this is true is this a bug?

No, by default BentoML runs sync endpoints with a max concurrency of 1. So any time there can only be at most 1 requests being processed. You can adjust it by setting

threads=N

parameter in the

service

decorator