This message was deleted.

might be more of a tensorflow configuration issue - but curious why I’m able to make the predict call on the keras model artifact directly but running into issues when going through the runner

@Alvin Mao Can you try bentoml.keras.save_model() instead of bentoml.tensorflow.save_model()?

@Alvin Mao, you may have to explicitly pass in the signatures of your model during

save_model

. https://docs.bentoml.org/en/latest/frameworks/tensorflow.html#saving-a-trained-model

gotcha, thanks Tim + Sean - I’ll give that a look unfortunately, I think our goal is to use the tensorflow option. specifically we’re looking to host the model in a production setting where latency matters and the documentation recommends Keras not being recommeded due to latency issues + memory leaks

What part of the documentation says that? Under the hood we configure a lot of different options for each framework. If there's a particular option which is not production worthy, would like to take a look to see if we are using that or not. but as Sean said I think you can get it working with tensorflow by potentially changing the signatures.

it's the header on this front:

Both bentoml.keras and bentoml.tensorflow support Keras models. bentoml.keras utilizes the native model format and will give a better development experience to users who are more familiar with Keras models. However, the native model format of Keras is not optimized for production inference. There are known reports of memory leaks during serving time at the time of BentoML 1.0 release, so bentoml.tensorflow is recommended in production environments. You can read bentoml.tensorflow documentation for more information.

You can also convert a Keras model to ONNX model and use bentoml.onnx to serve in production. Refer bentoml.onnx documentation and tensorflow-onnx (tf2onnx) documentation for more information.