hi everyone, I am trying to use bentoml/openllm library. I have a question regarding the minimum GPU requirements. as whenever I do openllm serve {model_name}, although the model should use 12 GB or 24 GB of GPU memory I find that the process is taking up to 72GB! the process name is venv/xxxx/bin/python. so if any help is appreciated why this high GPU utilization is happening and how to reduce it. Thank you in advance

This is likely due to it trying to reserve all available GPU memory for KV store - which caches intermediate states during inference, necessary for speed and throughput

do you know how can I stop such behavior as I need the GPU memory in other simultaneous operations @Chaoyu

yes there should be a config - @Jiang what’s the config for setting GPU utilization rate?

@Chaoyu yes that makes sense but like maybe I allocate like 5-7 extra GBs but does it make sense for a model of 12 GB to consume 72GB? @Jiang if you can assist with that please to how to control GPU utilization rate. Thanks you inadvance

Yes it’s normal behavior for LLM inference runtime to occupy all available memory by default for best performance. Most production use cases will be setup this way. You can configure it to use less but usually not recommend- it’s quite easy to run into OOM issues this way

@Chaoyu can you let me know how to control the occupation as I need to deploy other services on the same GPU and the openllm model utilizes it all, so I would like to know how to adjust the option and then test it

Are you running with Docker containers by any chance? or just by running

openllm serve

?

@Chaoyu You are correct. The inference engine vLLM will try to use more GPU memory as kv cache for longer context length, more concurrency and prefix caches. The default gpu_memory_utilization is 0.9. Tuning that config should do the trick.

@Chaoyu I am using Nvidia A100 but will check if MIG is supported. I am using a docker container yes and sometimes test with openllm serve to isolate any problems but the goal is to use docker. Nice thank you for providing this config will try it. But can you advise where to set this config? @Chaoyu @Jiang Like is it in a certain file or should be added in my docker-compose. Thanks

Hi there, there are two ways to approach with this: 1. If you just want to modify a specific parameters for a given model, then you can go directly to

$(openllm repo default)/bentoml/bentos

and modify the

openllm_config.yaml

configuration directly for the engine config. For example the configuration for llama3.2 is here:`/Users/work/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/llama3.2/11b-vision-instruct-1fbd/src/openllm_config.yaml` (note that this won’t persist once

openllm repo update

isused) 2. If you wish to persist these changes I recommend this and maintain a tree of your own modification of bentos (lmk if you need help with this). This option is more suitable when you have a suite of bentos that is derived from our main branch. You can then add the repo via

openllm repo add my-repo <https://github.com/aarnphm/my-openllm-repo-tree>

. The bento can the be used with

openllm run llama2 --repo my-repo

If you have any specific use case that requires more customisation then we are all ears 🙂

Hi @Aaron Pham thank you for the detailed help, really appreciated. Will start working and testing that and if other help needed will communicate with you. Thank you all for helping. 🙏

Lmk if there are anything else i dan help with. Feel free to ping me in this thread