Hi is it recommended to deploy bentoml service using kserve in k8s.

I think standalone kubernetes + docker deployment would be better and easier than kserve for most people

You can deploy on KServe any containerize app so it could be done with bentoml (my team and I have done it in the past). However, the few advantages of bentoml aren't worth using it on KServe in my opinion. Using FastAPI (or KServer Model Server) + Docker is a good match. Also, like Chaoyu said. KServe could be too much depending on your use case. You want something robust, cost effective and have a team with MLOps/DevOps. Then you can go for it. Otherwise, a managed service like bentocloud or google cloud run is more likely to be a better choice.

In my experience, another big hurdle many users have to go through in Kserve, besides the infra burden, is enforcing the Inference Protocol V2. It makes sense for tabular and classical ML models, but make everything else much harder (like LLM, embeddings, genAI, etc).

Since KServe 0.13 they did the integration of OpenAI protocol standards

Hi @Lucas I'm new for BentoML... So I'm curious about your opinion about this ->

However, the few advantages of bentoml aren't worth using it on KServe in my opinion.

Could I ask which advantages of bentoml will be affected when it is served with kserve?

you still get the advantage of dynamic batching, better serving performance (io serialization, model parallelization, etc), auto containerization - you will miss the fast cold-start, async task, multi-model distributed deployment, and some of the monitoring/management features for easy of use