This message was deleted.
# ask-for-helps
Slackbot
08/21/2023, 9:11 PMThis message was deleted.
j
Jian Shen Yap
08/21/2023, 9:15 PM1. Performance wise on the architecture level should be the same, since it is both BentoML Runner under the hood. The difference will be the implementation details of the runner.
2. I think batching is not available on OpenLLM's model afaik.
3. Yes
@Aaron Pham CMIIW
a
Aaron Pham
08/21/2023, 9:16 PM2. Batching is disabled for all OpenLLM runners
3. OpenLLM has option to mask directly to this multi-GPU assignment
--workers-per-resourceAaron Pham
08/21/2023, 9:17 PM1. OpenLLM Runners are more specialized comparing to
bentoml.transformersy
Yilun Zhang
08/22/2023, 1:33 PMThanks @Jian Shen Yap and @Aaron Pham for the quick response! Sorry I got pulled away and didn’t finish the reply.
So from your answers, if my end goal is to 1) host a LLM on a machine with multiple GPUs with bentoml, 2) enable dynamic batching, my option is to go with BentoML and create a custom runner class to define my own generation method, very much the same as hosting a standard transformers model, right? For the configuration and device mapping, I will just use the standard bentoml configuration yaml file.
a
Aaron Pham
08/22/2023, 2:34 PMSure, imho batching doesn’t help a lot when working with LLM that requires more than 1 GPU to load in. If you have multiple A100s then I think you should write your own custom Runner and define your own generation method.
I think if you want to run on smaller GPU or multiple GPUs that is not A10g and A100, having more than one instance of runner doesn’t help anyway, since you will probably need to use all of the GPUs in order to load the model
y
Yilun Zhang
08/22/2023, 2:35 PMWe do have a number of A10/100 GPUs, so each model is loaded on a single GPU, and potentially multiple instances on the same GPU, so that’s not really an issue. We are experimenting to maximize throughput right now. For example, with 1 40GB A100 GPU, I should be able to load 2 7b models instances in bfloat16,
a
Aaron Pham
08/22/2023, 2:37 PMyes
Aaron Pham
08/22/2023, 2:38 PMI think you can just use
openllm.Runner Copy code
runner = openllm.Runner("llama", model_id="meta-llama/Llama-2-7b-chat-hf") Copy code
runners:
llm-llama-runner:
resources:
<http://nvidia.com/gpu|nvidia.com/gpu>: [0]Aaron Pham
08/22/2023, 2:38 PMOr you can always go the route of implementing your own generation method
y
Yilun Zhang
08/22/2023, 2:39 PMThanks! I do want to have more control in the prompt construction by creating extra methods in the runner class, since the input itself isn’t full prompt but rather pieces of placeholders in the prompt.
a
Aaron Pham
08/22/2023, 2:39 PMI think this can leave to the client to do this
Aaron Pham
08/22/2023, 2:40 PMprompt construction shouldn’t be at the server level
Aaron Pham
08/22/2023, 2:40 PMbut it makes sense if you want to include the prompt into the server as well
Aaron Pham
08/22/2023, 2:40 PMI’m actually working on improving the client for openllm a bit to support prompt construction 😄
y
Yilun Zhang
08/22/2023, 2:41 PMCool, looking forward to more updates! But I think I will proceed with the older BentoML format for now so that I have better control
a
Aaron Pham
08/22/2023, 2:41 PMsounds good
y
Yilun Zhang
08/22/2023, 3:00 PMThank you for the help!
a
Aaron Pham
08/22/2023, 3:00 PMHappy to help! Please let me know if there are anything I can help with
🙌 1