This message was deleted.
# ask-for-helps
Slackbot
11/18/2022, 8:52 PMThis message was deleted.
a
Aaron Pham
11/19/2022, 12:56 AMHi @Jim Rohrer, can you send your service definition here?
j
Jim Rohrer
11/19/2022, 1:17 AMFor sure:
Copy code
class BEiTPreprocessor(bentoml.Runnable):
SUPPORTED_RESOURCES = ("cpu","<http://nvidia.com/gpu|nvidia.com/gpu>")
SUPPORTS_CPU_MULTI_THREADING = True
def __init__(self):
self.beit_processor = BeitFeatureExtractor.from_pretrained(os.path.join(BASE_PATH, "beit_preprocessor"))
@bentoml.Runnable.method(batchable=True, batch_dim=0)
def __call__(self, input_image: List["Image.Image"]):
print(type(input_image[-1]))
print(input_image)
try:
print(input_image.shape)
except Exception as e:
print(e)
beit_input = self.beit_processor(images=input_image, return_tensors="pt").to(device)
return beit_input
beit_preprocessor_runner = bentoml.Runner(BEiTPreprocessor, name="beit_preprocessor", max_batch_size=4, max_latency_ms=3000)
svc = bentoml.Service('gun_model_service', runners=[beit_preprocessor_runner])
@svc.api(input=Image(), output=JSON())
async def classify(input_image: Image) -> list:
beit_inputs = await beit_preprocessor_runner.async_run([input_image])
print(beit_inputs)
return {"batch_size": len(beit_inputs["pixel_values"])}Jim Rohrer
11/21/2022, 7:19 PMSo I think I figured out that if batching is enabled, the runner needs to return a list with the same number of values as came into the runner, but I still can't seem to get it to actually batch the image inputs. I only ever get a single item.
a
Aaron Pham
11/21/2022, 8:21 PMHmm transformers should by default support multiple outputs. Our runner implementation simply returns what transformers output gives.
j
Jim Rohrer
11/21/2022, 11:13 PMthat actually threw me....when you pass the BeitFeatureExtractor a list of images, the return value looks like
{'pixel_values': [<tensor>, <tensor>]}[{'pixel_values': <tensor>}, {'pixel_values': <tensor>}]Jim Rohrer
11/21/2022, 11:13 PMso it kept returning a single dict which threw Bento's batcher