This message was deleted.

@Amar Ramesh Kamat can you share your code?

Let me try to share as much as I can 🙂

sounds good

import tensorflow as tf
import bentoml
=
service = 'foo'
model_name = 'bar'
model_dir = "models/" + model_name

model = tf.saved_model.load(model_dir)
    
bentoml_model = bentoml.tensorflow.save_model(
    service,
    model,
    signatures={"__call__": {"batchable": False}}
)

Here, I have a TF-2.x model on my local disk. I am loading it and then saving it via bento wrapper

How do you run the inference? How does the expected input looks like?

Here is how the service.py looks like

import numpy as np
import bentoml
import tensorflow as tf
from PIL import Image
from <http://bentoml.io|bentoml.io> import JSON
from bentoml._internal.types import JSONSerializable

model_name = 'bar'
# tried model = bentoml.model.get(model_name + ":latest") too
model = bentoml.tensorflow.get(model_name + ":latest")
runner = model.to_runner()

svc = bentoml.Service(model_name, runners=[runner])

@svc.api(input=JSON(), output=JSON())
async def predict(json_obj: JSONSerializable) -> JSONSerializable:
    retur await runner.async_run([json_obj])

Currently our tensorflow v2 implementation only accept list/tensor/ndarray inputs. Could you do the custom processing outside the model?

We can. Just that its a change to the existing system.

Currently our tensorflow v2 implementation only accept list/tensor/ndarray inputs.

That’s what I expected.

Actually our implementation also expected model outputs to be ndarray/tensor. Because in our own pass experience the inputs and outputs of a tensorflow model are usually tensor-like objects. Do you think there are a lot of usage for non tensor inputs/outputs?

Yes. Majority of the models we have is deployed via TF Serving and the easiest way to do some custom pre-post processing was to wrap the model like this.

Could you do a couple of custom Runners to do your pre- and post-processing?

Amar, Thanks for your input. It's very helpful for us to decide which features we should support. If possible, could you point me some resource (tutorial etc.) for this kinds of usage so we can study and try to add support for it?

Are you referring to custom wrappers?

Not exactly. You can take your already initialized model (when saving the models to BentoML), add a function to it that calls your actual predict or call functions. And then you can simply run code before and after the model inference. For example I had a response from my inference that wasn't compatible with BentoML's expectations so thats what I did:

def pytorch_predict(self, d):
    return [x.pandas().xyxy for x in self(d).tolist()]

model.custom_predict = types.MethodType(pytorch_predict, model)

Thanks @Yakir Saadia @larme (shenyang) @Jim Rohrer . Will try out these suggestions and get back