This message was deleted.

Hi Yilun, I highly recommend you to check out the new project we just dropped here: https://github.com/bentoml/OpenLLM

For the BentoML side, you can check out https://docs.bentoml.org/en/latest/guides/scheduling.html Let say that you have 4 GPUs available. Given that you only want 1 instance of the runner, the following configuration can be used:

runners:
  resources:
    <http://nvidia.com/gpu|nvidia.com/gpu>: [0,1,2,3]
  workers_per_resources: 0.25

Hey @Chaoyu, thanks for the direction, I briefly looked at it but haven’t moved forward with experimentation yet as I see there are limited number of model types that are supported and not sure how it goes with model architectures not mentioned there. It seems to be advised to have support for any models, does it really mean it? I guess I can give it a shot today to see how things goes!

what models are you looking for?

Hey @Aaron Pham, thanks for the useful resource, I will definitely check it out! Another issue I found while using LLM with BentoML is, if some models have extra remote code, it sometimes cause issues while hosting in Bentoml where it says the model can’t be loaded with any available model architecture in transformers library. In my specific case, falcon-7b-instruct worked fine but when I tried to host falcon-40b-instruct in 8-bit, I saw this issue. Checking both bentoml model repository, I didn’t see the remote python code being copied over, so it remains mystery to me why 7b works but not 40b one.

hmm the support from bentoml with

trust_remote_code

is pretty limited atm.

I see, how about in openllm? Is there good support on remote code?

I wrote support for chatglm, falcon and dolly-v2, which essentially relies on

trust_remote_code

. OpenLLM do have support for

trust_remote_code

so I want to optimistically say that it is well supported 😄

That’s very nice to hear, I will give it a try!

Yes we do have 8 bit support. OpenLLM models can already load in 8bit. Note that 8bit is not a silver bullets 🙂

Got it, thanks for sharing the progress and roadmap!

Currently, OpenLLM provides a Runner integration with BentoML, which allows you to easily run LLM with other models with BentoML. This means you will have full control in terms of resource allocation for the LLM. If you just want to serve the LLM, then I think it makes sense to just use

openllm start

and let OpenLLM manage the server for you. (esp for the frontend UI, like react or a gradio app)

From the doc, it seems like the hosting is 1-port-to-1-model, is it correct? If I want to host multiple, then I will need to have them on different ports.

That is correct. We can possibly support multiple models/port, but it will involve more specific configuration settings, (def achievable)

Got it, thanks! I think independent hosting, especially for LLM can be more natural as it’s probably unlikely to host multiple of them. Just wondering, in this case if I’m hosting with openllm, I guess (at least for now) there’s not support for multi-instance and dynamic batching etc. right?

Yes, currently I overwrite the configuration set for dynamic batching, but obv we can always support it later.

Makes sense, thank you for the explanation!