From my understanding, since BentoML 1.2, instead of using Runners for the models, we're supposed to use another Service. We can then define the dependency between the services and add

batchable=True

to the service with the model. However, one thing I noticed is that any

@api

method with

batchable=True

can only accept one argument and it doesn't work with a dictionary. Back then, I needed to implement my own Runnable as suggested by BentoML team to be able to pass batchable dictionaries and I tried the same with the new organization of the code, but it doesn't work now because of this limitation. As reference, this was my custom Runnable:

class PyTorchModelWithTensorDictRunnable(bentoml.Runnable):
    SUPPORTED_RESOURCES = ("<http://nvidia.com/gpu|nvidia.com/gpu>", "cpu")
    SUPPORTS_CPU_MULTI_THREADING = True

    def __init__(self, model_name: str):
        super().__init__()
        if torch.cuda.is_available():
            self.device_id = "cuda"
            # by default, torch.FloatTensor will be used on CPU.
            torch.set_default_tensor_type("torch.cuda.FloatTensor")
        else:
            self.device_id = "cpu"
        self.model = bentoml.pytorch.load_model(model_name, device_id=self.device_id)
        # We want to turn off dropout and batchnorm when running inference.
        self.model.train(False)

    @bentoml.Runnable.method(batchable=True)
    def __call__(self, **batch):
        # Our model accepts a dictionary of Tensor as input, but we use ``**``
        # to deconstruct it to keyword arguments, so they can be batched rightly.
        # It works only if every parameter is "batchable" such as torch.Tensor/np.ndarray.
        # Move the tensors to the target device and pass it to the model as a dict.
        for i, (k, tensor) in enumerate(batch.items()):
            if i == 0:
                observe_adaptive_batch_size_metric(tensor.shape[0])
            batch[k] = <http://tensor.to|tensor.to>(self.device_id)
        with torch.inference_mode():
            return self.model(batch)

What is the best alternative now? For now, I'm passing a tuple and then converting it back to

dict

, but it's far from perfect. Especially because I want to have a single codebase for the deployment that will be able to deploy different torch models (that might use different inputs).

@Frost Ming could you help take a look? @Fernando Camargo sorry about the delay in getting back to you, I have been sick the past few days

No problem, @Chaoyu. To give an update on that, the only thing that worked for me was sending a list of Pydantic objects. Then, on the other side I do the preprocessing and convert into tensors. I also tried: • Sending a tuple of lists, but it ended up becoming a huge tuple instead of batching at the elements of the tuple. The batch_dim was set to (

0, 0)

, but still it didn't work. • Sending a list of tuples, but there was a ReadError in the request between the services.

@Fernando Camargo sorry for late reply, actually we have a guide for handling multiple arguments in batchable methods: https://docs.bentoml.com/en/latest/guides/adaptive-batching.html#handle-multiple-parameters You may need to extract the tensors from the list and stack them yourself, tell me if that meets your requirements