Wow, I'm realizing this question could be expressed as "how can you write a batch-enabled Runnable whose return type represents a batch not as a numpy array?"

You can return something like

class CustomeRunner:
    def predict(self):
        ...
        return [image1, image2], [score1, score2]

A batch in bentoml could be also a list of anything pickle serializable. And bentoml supports returning multiple outputs by returning a tuple like above

When returning lists, the batch_dim should be 0

Okay, I created a simple service with a runner meant to work the same as yours:

It runs, but I'm not sure how to test it. I opened 3 OpenAPI pages at

<http://localhost:3000>

and quickly clicked on the "execute" button on all three pages. With these inputs: Inputs:

{
  "input": 0
}

{
  "input": 1
}

{
  "input": 2
}

Output:

{
  "output": [
    "image0",
    0
  ]
}

It looks like the batching is not happening. I tried running it with the

--production

flag as well with the same result.

Oh also, did you mean to write

[image1, score1], [image2, score2]

in your example?

Weird, I can't edit old messages. I meant to say here that all three requests give the same output. That as well as some print statements seem to show that batching isn't happening. Is there an easy way to force batching to occur? I added

time.sleep(2)

into the

/predict

handler in an attempt to get the server to hang long enough for me to click all 3 "execute" buttons in the browser windows I had open, but it didn't seem to work.

It should be

return [image1, image2], [score1, score2]

In Python it will return a

tuple([image1, image2, ...], [score1, score2, ...])

bentoml will treat them as multiple output