This message was deleted BentoML #ask-for-help

Join Slack

This message was deleted.

# ask-for-help

Slackbot

07/28/2023, 10:07 PM

This message was deleted.

Chaoyu

07/31/2023, 2:25 AM

Hi @Visaal Ambalam, did you specify the batchable option and batch dim when saving the model? or are you using your own Runnable implementation here?

Visaal Ambalam

07/31/2023, 2:31 PM

I specified the batchable option with the default batch dim (0) when saving the model like this. The model itself’s

infererence_v2

method only takes 1 pytorch image tensor as an argument at a time, so I’m not sure if the underlying BentoML batching code knows how to batch image tensors.

Copy code

pretrained_model: DonutModel = DonutModel.from_pretrained(pretrained_model_path)

    name = "id_document_classifier_model"
    model = bentoml.pytorch.save_model(
        name,
        pretrained_model,
        signatures={
            "inference_v2": {
                "batchable": True,
                "batch_dim": 0,
            }
        },
        custom_objects={"encoder": pretrained_model.encoder, "decoder": pretrained_model.decoder},
    )
    runner = model.to_runner()
    runner.init_local()
    out = runner.inference_v2.run(image_tensors=img_tensor)["predictions"][0]

Visaal Ambalam

07/31/2023, 2:34 PM

It says here that it supports batching `torch.Tensor`s, but it’s confusing because the example below implements a custom runner to batch the tensors, so I’m not sure what the expected functionality is. I also couldn’t find where an exhaustive list of batchable types were on the documentation website or github code were.

Open in Slack

Previous Next