https://bentoml.com logo
Join Slack
Powered by
Trying to use openllm to run the llama3.1-8b model...
# ask-for-help
p
Trying to use openllm to run the llama3.1-8b model.
c
@Aaron Pham could you help take a look?
a
Yep im doing a full cleanup atm
r
Hi I am the new one using openllm. now, I am running openllm serve gemma:2b or extra command, but it keeps failing. and it loops. the error msg is like this /.openllm/venv/1004407653884267918/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py", line 651, in init_app_state chat_template_content_format=args.chat_template_content_format, AttributeError: 'Namespace' object has no attribute 'chat_template_content_format' please help me
a
Yes this has to do with the breaking change in vllm 0.7. I’m in the process of updating the bentos promptly
r
Thanks for your reply. so you mean that is due to the version of vllm or openllm?
a
It is vllm, but also at openllm-models
No need to upgrade openllm
r
openllm uses vllm as backends, i think.
a
Hi there, please update with the latest model repo via 
openllm repo update
thanks. I have updated to the latest vllm version
r
thanks. now i have AssertionError: cannot find version file in /home/Dmytro/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/gemma/2b-instruct-fp16-b322 when running openllm run gemma:2b What is wrong? Thanks.
a
I cut a new openllm releases that remove this assertion check, the models repo looks ok to me, but it seems like there are a bug for running gemma overall with vLLM
Copy code
File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 377, in load_model
    model = _initialize_model(vllm_config=vllm_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 119, in _initialize_model
    return model_class(vllm_config=vllm_config, prefix=prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/models/gemma2.py", line 412, in __init__
    self.model = Gemma2Model(vllm_config=vllm_config,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 149, in __init__
    old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/models/gemma2.py", line 261, in __init__
    self.start_layer, self.end_layer, self.layers = make_layers(
                                                    ^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 555, in make_layers
    [PPMissingLayer() for _ in range(start_layer)] + [
                                                     ^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 556, in <listcomp>
    maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/models/gemma2.py", line 263, in <lambda>
    lambda prefix: Gemma2DecoderLayer(
                   ^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/models/gemma2.py", line 187, in __init__
    self.self_attn = Gemma2Attention(
                     ^^^^^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/vllm/model_executor/models/gemma2.py", line 148, in __init__
    config.interleaved_sliding_window is not None)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.openllm/venv/898325974958319280/lib/python3.11/site-packages/transformers/configuration_utils.py", line 211, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Gemma2Config' object has no attribute 'interleaved_sliding_window'
This happens with gemma2:9b, but gemma2:2b should work
lmk if there are anything I can help with
r
Thanks for you help. But could you guide me how you removed assertion error and I can run gemma:2b model successfully?
a
if you update the latest to latest openllm 
pip install -U openllm
and 
openllm repo update
it should work (i tested with this on my end)
r
Thanks a lot. it works on my end. 👍
🏁 1
p
Thanks for the fix. It works now. Is there a way to use other gpu other than the the first one? I set the variable CUDA_VISIBLE_DEVICES=3 but openllm doesn't respect the setting.
a
I’m pretty sure that we don’t set any specific GPU envvar, so this might have to be with vLLM. Can you send me the nvidia-smi display?
p
The CUDA_ViSIBLE_DEVICES works when I use vllm directly to run a llm model. Here is the output from the nvidia-smi: root@llmserver:~# nvidia-smi Mon Feb 10 163643 2025 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.216.01 Driver Version: 535.216.01 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA L40S Off | 000000000100.0 Off | 0 | | N/A 29C P0 78W / 350W | 41408MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 1 NVIDIA L40S Off | 000000002100.0 Off | 0 | | N/A 24C P8 31W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 2 NVIDIA L40S Off | 000000004100.0 Off | 0 | | N/A 23C P8 32W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 3 NVIDIA L40S Off | 000000006100.0 Off | 0 | | N/A 22C P8 31W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 4 NVIDIA L40S Off | 000000008100.0 Off | 0 | | N/A 29C P0 77W / 350W | 41296MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 5 NVIDIA L40S Off | 00000000A100.0 Off | 0 | | N/A 24C P8 32W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 6 NVIDIA L40S Off | 00000000C100.0 Off | 0 | | N/A 23C P8 32W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 7 NVIDIA L40S Off | 00000000E100.0 Off | 0 | | N/A 23C P8 32W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 447448 C ...venv/1682988277965704177/bin/python 41394MiB | | 4 N/A N/A 29534 C ...p/home/vllm/deepseek-ai/bin/python3 41282MiB | +---------------------------------------------------------------------------------------+
Here are the process listing for openllm: root@llmserver:~# ps uax|grep openllm openllm 447358 0.0 0.0 33380 3340 ? Ss 11:55 0:00 /bin/bash /var/local/home/openllm/serve-llama3.1 openllm 447359 0.0 0.0 205336 93196 ? S 11:55 0:00 /var/local/home/openllm/openllm/bin/python3 /var/local/home/openllm/llama/bin/openllm serve llama3.1:8b --port 37090 openllm 447369 0.0 0.0 2932348 128500 ? Sl 11:55 0:07 /var/local/home/openllm/.openllm/venv/1682988277965704177/bin/python -m bentoml serve llama3.1:8b-instruct-189d --port 37090 openllm 447448 2.2 0.3 89155856 5792092 ? Ssl 11:55 6:14 /var/local/home/openllm/.openllm/venv/1682988277965704177/bin/python -m _bentoml_impl.worker.service service:VLLM --fd 4 --service-name llama3.1 --backlog 2048 --worker-id 1 --ssl-version 17 --ssl-ciphers TLSv1 --wer-env [{"CUDA_VISIBLE_DEVICES": "0"}]
I see that it uses CUDA_VISBLE_DEVICES but it defaults to 0.
a
cc @Jiang
j
openllm did nothing with this. @Frost Ming Does bentoml inject this ?
f
Yes unless 
BENTOML_DISABLE_GPU_ALLOCATION
is set
j
Got, will set this in OpenLLM
p
Setting the BENTOML_DISABLE_GPU_ALLOCATION=true works. The CUDA_VISIBLE_DEVICES is respected now by openllm. Thanks for the info.
Is there a way to tell openllm to use multiple GPUs? I need to use multiple GPUS in order to run the 70B parameter models. I don't see any option in openllm to do that.
a
you should be able to run the 70B models with openllm assuming you have two GPUs. it should already set the TP parameters accordingly
p
How do set the TP parameters? Can you give me some detail steps? Thanks.
a
you can just set CUDA_VISIBLE_DEVICES=0,1 openllm should already set the TP
p
Is there a way to pass these options in vllm to openllm?
--dtype=half
i have older GPUs that i want to use to run the llama3.1 8B model
this is the error i'm getting when trying to use openllm to run the llama3.1 8B model using the four gtx 1080 ti with 11GB VRAM each: ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your NVIDIA GeForce GTX 1080 Ti GPU has compute capability 6.1. You can use float16 instead by explicitly setting the`dtype` flag in CLI, for example: --dtype=half.
Open in Slack
PreviousNext
Powered by