Hi BentoML team, I have an issue with regards to adaptive batching. I managed to get adaptive batching to work with a simple example (get payload size of more than 1). However, for my use case I need to use this template from documentation: https://docs.bentoml.com/en/latest/guides/adaptive-batching.html#handle-multiple-parameters In my service I log the size of the payload I receive and it is always of size 1. I feel like I literally did what documentation tells us but I cannot get adaptive batching to kick in. Maybe someone could have a look at my code and see if I do something wrong?

import bentoml
from typing import Any
from pydantic import BaseModel


class BatchInput(BaseModel):
    words: list[str]
    boxes: list[list[int]]

# BentoML service definition
@bentoml.service(resources={"cpu": "2", "memory": "8Gi"})
class BatchService:
    @bentoml.api(
        batchable=True,
        max_batch_size=8,
        max_latency_ms=500000
    )
    def predict(self, payload: list[BatchInput]) -> list[dict[str, Any]]:
        """
        Batchable prediction API for CPU inference.

        :param payload: A dictionary containing 'words_batch' and 'boxes_batch'.
        :return: A list of prediction results.
        """
        print(f"Payload length: {len(payload)}")
        results = [{"prediction": 0}]

        return results


@bentoml.service
class MyService:
    batch = bentoml.depends(BatchService)

    @bentoml.api
    def generate(self, words: list[str], boxes: list[list[int]]) -> list[dict]:
        result = self.batch.predict([BatchInput(words=words, boxes=boxes)])
        return result

this is my requirements file for this vanilla service:

bentoml==v1.2.16
pydantic==2.8.2
pydantic-core==2.20.1

This is the logs from the service:

Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.014ms (trace=b78e72aa187dceaa1e2f3980e6d70067,span=149814188f09b042,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:41034 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 354.766ms (trace=fc3469800666fcf7c650366045f7fb0a,span=6c36c22c94966e58,sampled=0,service.name=MyService)
Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.324ms (trace=76a7a9fbb7f2990e1b1afacb44b3778e,span=cc953adf8b756233,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:40984 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 354.662ms (trace=b12d9c4bd3fad69c3403b3c1d14ed9ae,span=573e2be2754131b9,sampled=0,service.name=MyService)
Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.016ms (trace=b20b682f68eb44ae71d9a648098b71df,span=8132ebc6a388fc40,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:40996 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 353.710ms (trace=dc65f1ea9689c30be4e7a55c36c16d53,span=6724f8c932789935,sampled=0,service.name=MyService)
Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.378ms (trace=4d9dd69e32ca474ce0b112b41ee5bd42,span=19e63ffe6bd1f456,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:40950 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 354.151ms (trace=8a46385e7956cc17c1d683f85178d7ca,span=3ef452190fcdc8f7,sampled=0,service.name=MyService)
Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.228ms (trace=5b03893b1f1005603327c0f2adba11e2,span=2c577776711457f4,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:40662 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 354.885ms (trace=3bcc6af0cec0fb0ad6232d175efbf110,span=f7113b9ba264c2ff,sampled=0,service.name=MyService)
Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.148ms (trace=2762c90332c41ccc21e7402319dde798,span=3776747126a60fb8,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:40646 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 355.173ms (trace=605483d5b1d7b2e1c9e5f55dbacedd59,span=1ab852360d32525b,sampled=0,service.name=MyService)
Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.298ms (trace=4b5a26860bbb3ae4127f4af4431a897d,span=27d0037d037f1348,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:41052 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 355.412ms (trace=89396667e32e8c2123de4592b458f272,span=291ca25b98277ade,sampled=0,service.name=MyService)
Payload length: 1
2024-09-20T08:52:34+0000 [INFO] [service:BatchService:1] 127.0.0.1:49440 (scheme=http,method=POST,path=/predict,type=application/vnd.bentoml+pickle,length=12219) (status=200,type=application/json,length=33) 2.204ms (trace=741e3a698f87cc69f8d35a4bb9cb4a4d,span=c4dc1ede4204f5f0,sampled=0,service.name=BatchService)
2024-09-20T08:52:34+0000 [INFO] [entry_service:MyService:1] 172.17.0.1:41068 (scheme=http,method=POST,path=/generate,type=application/json,length=17753) (status=200,type=application/json,length=18) 355.576ms (trace=e50a68f03f4adede1304cac20e64fb68,span=3c839d255d7952cc,sampled=0,service.name=MyService)
Payload length: 1

I test my service with

bentoml build --containerize

and then run

docker run -p 3000:3000 my_service:tag

Thanks for spotting this! we will update our documentation