Hallo! I am across new BentoVLLM how is this different from OpenLLM. I used OpenLLM for serving llm before now there is new BentoVLLM so am confused is this new one better??

Hi @Suhas - BentoVLLM is an example project providing a bare minimum structure for serving LLM with BentoML and VLLM. It is easy to customize if you’re familiar with BentoML, and allow advanced users to build on top, e.g. composing multiple LLMs or chain LLM calls with smaller language models or other type of models. OpenLLM on the other side is fully packaged solution for self-hosting LLM. If you just want to run an open source LLM over an OpenAI compatible endpoint, it is super easy to do so. You can customize OpenLLM’s behavior via parameters, configs, or providing fine-tuned model weights.

So there is no differences in performance within library right?

yes there should be no difference in performance

This feature would great to have

The upcoming Openllm 0.5 will be upgraded to 1.2 😄

Will there documentation about openllm about services, because now old documents have been replaced with vllm tutorials.