https://bentoml.com logo
Join Slack
Powered by
Hi!! I jsut updated my OpenLLM version and now I c...
# ask-for-help
z
Hi!! I jsut updated my OpenLLM version and now I cant run my model of choice
Copy code
openllm start argilla/CapybaraHermes-2.5-Mistral-7B
how do i add my model into openllm
j
@Jiang
j
@Zak Ozbourne Here is an example to contribute a model to the official repo of OpenLLM. You can also maintain your own repo. https://github.com/bentoml/openllm-models/pull/1
z
Thank you Is there are slightly more indepth guide on how to add your own models without doing a pull request? I have cloned forked the model repo, modified the recipe.yaml, added the new repo as the default When trying to make.py this is what I get
Copy code
openllm@openllm:~/.openllm/repos/github.com/zak23/zaks-openllm-models/main/source$ BENTOML_HOME=$(openllm repo default)/bentoml/bentos python3 make.py mistral:capybara-7b-fp16
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  |     return _run_code(code, main_globals, None,
  |   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
  |     exec(code, run_globals)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/bentoml/__main__.py", line 4, in <module>
  |     cli()
  |   File "/home/openllm/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
  |     return self.main(*args, **kwargs)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
  |     rv = self.invoke(ctx)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
  |   File "/home/openllm/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
  |     return ctx.invoke(self.callback, **ctx.params)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
  |     return __callback(*args, **kwargs)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/bentoml_cli/utils.py", line 362, in wrapper
  |     return func(*args, **kwargs)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/bentoml_cli/utils.py", line 333, in wrapper
  |     return_value = func(*args, **kwargs)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
  |     return f(get_current_context(), *args, **kwargs)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/bentoml_cli/utils.py", line 290, in wrapper
  |     return func(*args, **kwargs)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
  |     return f(get_current_context(), *args, **kwargs)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/simple_di/__init__.py", line 139, in _
  |     return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  |   File "/home/openllm/.local/lib/python3.10/site-packages/bentoml_cli/bentos.py", line 368, in build
  |     build_config = BentoBuildConfig.from_yaml(f)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/bentoml/_internal/bento/build_config.py", line 869, in from_yaml
  |     return bentoml_cattr.structure(yaml_content, cls)
  |   File "/home/openllm/.local/lib/python3.10/site-packages/cattrs/converters.py", line 334, in structure
  |     return self._structure_func.dispatch(cl)(obj, cl)
  |   File "<cattrs generated structure bentoml._internal.bento.build_config.BentoBuildConfig>", line 66, in structure_BentoBuildConfig
  |     if errors: raise __c_cve('While structuring ' + 'BentoBuildConfig', errors, __cl)
  | cattrs.errors.ClassValidationError: While structuring BentoBuildConfig (1 sub-exception)
  +-+---------------- 1 ----------------
    | cattrs.errors.ForbiddenExtraKeysError: Extra fields in constructor for BentoBuildConfig: envs
    +------------------------------------
Traceback (most recent call last):
  File "/home/openllm/.openllm/repos/github.com/zak23/zaks-openllm-models/main/src/make.py", line 89, in <module>
    subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'bentoml', 'build', '/tmp/tmph2e1am18', '--version', 'capybara-7b-fp16-e5ab']' returned non-zero exit status 1.
This is what I have in my recipe.yaml
Copy code
"mistral:capybara-7b-fp16":
  project: vllm-chat
  service_config:
    name: mistral
    traffic:
      timeout: 300
    resources:
      gpu: 1
      gpu_type: nvidia-rtx-3090
  engine_config:
    model: argilla/CapybaraHermes-2.5-Mistral-7B
    max_model_len: 4096
    enforce_eager: true
    dtype: half
  chat_template: mistral-instruct
  extra_labels:
    openllm_alias: 7b
    model_name: argilla/CapybaraHermes-2.5-Mistral-7Bv0.1
Copy code
openllm@openllm:~$ echo BENTOML_HOME=$(openllm repo default)
BENTOML_HOME=/home/openllm/.openllm/repos/github.com/zak23/zaks-openllm-models/main
j
What bentoml version are you using? How about 
pip install -U bentoml
then try again?
We are planning to add a dockerfile and Github action to automatically build bentos from 
src
in the next step. Do you think it will help?
z
ok, thank you for your response, it appears i was stuck in some sort of dependency hell, but i have gotten out of it
Copy code
openllm@openllm:~/.openllm/repos/github.com/zak23/zaks-openllm-models/main/src$ BENTOML_HOME=$(openllm repo default)/bentoml/bentos python3 make.py "mistral:7b-capybara-fp16"
INFO: Adding current BentoML version to requirements.txt: 1.3.0
INFO: Locking PyPI package versions.

██████╗ ███████╗███╗   ██╗████████╗ ██████╗ ███╗   ███╗██╗
██╔══██╗██╔════╝████╗  ██║╚══██╔══╝██╔═══██╗████╗ ████║██║
██████╔╝█████╗  ██╔██╗ ██║   ██║   ██║   ██║██╔████╔██║██║
██╔══██╗██╔══╝  ██║╚██╗██║   ██║   ██║   ██║██║╚██╔╝██║██║
██████╔╝███████╗██║ ╚████║   ██║   ╚██████╔╝██║ ╚═╝ ██║███████╗
╚═════╝ ╚══════╝╚═╝  ╚═══╝   ╚═╝    ╚═════╝ ╚═╝     ╚═╝╚══════╝

Successfully built Bento(tag="mistral:7b-capybara-fp16-e5ab").

Next steps:

* Deploy to BentoCloud:
    $ bentoml deploy mistral:7b-capybara-fp16-e5ab -n ${DEPLOYMENT_NAME}

* Update an existing deployment on BentoCloud:
    $ bentoml deployment update --bento mistral:7b-capybara-fp16-e5ab ${DEPLOYMENT_NAME}

* Containerize your Bento with `bentoml containerize`:
    $ bentoml containerize mistral:7b-capybara-fp16-e5ab

* Push to BentoCloud with `bentoml push`:
    $ bentoml push mistral:7b-capybara-fp16-e5ab
how do i get this model into my openllm without doing a pull request. I would like to experiment with many different models on my 3x RTX 3090s
j
You can add your own repo like this 
openllm repo add dev <https://github.com/bentoml/openllm-models@dev>
. Then just 
openllm repo update
. Just like apt.
z
I have that set up It's just a fork of the main model repo 
<https://github.com/zak23/zaks-openllm-models|https://github.com/zak23/zaks-openllm-models>
I have edited the recipe.yaml I run the make.py and it says it's all ok (as above) What is the next step?
j
Oh, 
make
will build bentos and put under 
(REPO_ROOT)/bentoml/bentos
. Then just need to commit the built bentos and push to your repo.
Copy code
git add .
git commit -m "add mistral:7b-capybara-fp16"
git push
then
Copy code
openllm repo add zak23 <https://github.com/zak23/zaks-openllm-models> # only need to be add once
openllm repo update
then you should be able to find the 
mistral:7b-capybara-fp16-e5ab
in 
openllm model list
z
thank you for all your help, I believe I am very close to cracking this, but I am getting an error when trying to load the model
Copy code
bentobox@bentobox:~/zaks-openllm-models/bentoml/bentos$ openllm model list
model              version               repo     required GPU RAM    platforms
-----------------  --------------------  -------  ------------------  -----------
capybarahermes2_5  capybarahermes2_5:7b  default  24G                 linux
bentobox@bentobox:~/zaks-openllm-models/bentoml/bentos$ openllm hello
  Detected Platform: linux
  Detected Accelerators:
   - NVIDIA GeForce RTX 3090 24GB
? Select a model capybarahermes2_5  default  Yes
? Select a version capybarahermes2_5:7b  Yes
? Select an action 0. Run the model in terminal


$ export BENTOML_HOME=/home/bentobox/.openllm/repos/github.com/zak23/zaks-openllm-models/main/bentoml
$ source /home/bentobox/.openllm/venv/1400447886094987151/bin/activate
$ bentoml serve capybarahermes2_5:7b-capybara-fp16-36b8 --port 39103
Model server started 4727
Model loading...
Error: [serve] `serve` failed: Failed to load bento or import service 'capybarahermes2_5:7b-capybara-fp16-36b8'.
If you are attempting to import bento in local store: 'Attribute "7b-capybara-fp16-36b8" not found in module "capybarahermes2_5".'.
If you are importing by python module path: 'Failed to import module "service": cannot import name 'Schema' from 'pydantic' (/home/bentobox/.openllm/venv/1400447886094987151/lib/python3.10/site-packages/pydantic/__init__.py)'.
j
Seems like a fastAPI version error: https://github.com/gradio-app/gradio/issues/8877
You may need to lock 
fastapi==0.88.0
in the 
requirements.txt
and make +push+update again
z
where would i put that to make it?
j
src/vllm-chat/requirements.txt
z
Just when I think I am making progress 😅
Copy code
INFO: Adding current BentoML version to requirements.txt: 1.3.0
INFO: Locking PyPI package versions.
  × No solution found when resolving dependencies:
  ╰─▶ Because fastapi==0.88.0 depends on starlette==0.22.0 and bentoml==1.3.0 depends on starlette>=0.24.0, we can conclude that bentoml==1.3.0 and fastapi==0.88.0 are incompatible.      
      And because you require bentoml==1.3.0 and fastapi==0.88.0, we can conclude that the requirements are unsatisfiable.
Error: [bentos] `build` failed: Failed to lock PyPI packages: Command '['/usr/bin/python3', '-m', 'uv', 'pip', 'compile', '/tmp/tmpvlxb90ndbentoml_bento_capybarahermes2_5/env/python/requirements.txt', '--allow-unsafe', '--no-header', '--output-file=/tmp/tmpvlxb90ndbentoml_bento_capybarahermes2_5/env/python/requirements.lock.txt', '--resolver=backtracking', '--no-annotate', '--quiet']' returned non-zero exit status 1.
@Jiang any tips?
j
oh. Let's try to figure out a working dependency versions for your case. First let's change "fastapi==0.88.0" back to "fastapi", then
Copy code
cd src/vllm-chat
pip install -r requirements.txt -U   # to upgrade all the dependencies
bentoml serve .  # you can validate the bento before make and push
If you get the same exception 
cannot import name 'Schema' from 'pydantic'
, run 
pip freeze
and paste the output here
z
working on it now
j
any updates?
z
it seems to be downloading the model
j
nice
z
I did 
cd src/vllm-chat
pip install -r requirements.txt -U
bentobox@bentobox:~/zaks-openllm-models/src$ bentoml serve .
but it said: 
Error: [serve] serve failed: Failed loading service from path /home/bentobox/zaks-openllm-models/src. When loading from a path, it must be either a Bento containing bento.yaml or a project directory containing bentofile.yaml
so then I did: 
bentobox@bentobox:~/zaks-openllm-models/src/vllm-chat$ bentoml serve .
and it loaded `model='Qwen/Qwen2-7B-Instruct-AWQ`which is not my model so then i thought maybe... and i ran 
bentobox@bentobox:~/zaks-openllm-models/src$ BENTOML_HOME=/home/bentobox/zaks-openllm-models/bentoml/bentos python3 make.py "mistral:7b-capybara-fp16"
which then it succeeded! So then I ran: 
bentobox@bentobox:~/zaks-openllm-models/bentoml/bentos/bentos/capybarahermes2_5/7b-capybara-fp16-e953$ bentoml serve .
I know i have the path a little wrong here It downloaded the model and I could use it on http://192.168.1.150:3000/#/
so now i am cleaning up some dirs, re-making the bento, pushing to my repo, then will update my repo through openllm and try run the model
👍 1
j
Even though the loaded hf model is not correct, at least the code and dependencies works.
z
ok, this seems to be all working really well. Except its throwing lots of 
[INST] Hello! How can I help. [/INST]
Which I think is a chat template problem
j
Which hf model are you using?
It should be an instruct model instead of a text model
z
argilla/CapybaraHermes-2.5-Mistral-7B
Copy code
<https://huggingface.co/argilla/CapybaraHermes-2.5-Mistral-7B>
j
I see. Finally it fine-tuned from https://huggingface.co/mistralai/Mistral-7B-v0.1 , which is not an instruct model
z
oh yea. I am just trying to get back to where i was before the OpenLLM update. I had a few projects running on 
openllm start argilla/CapybaraHermes-2.5-Mistral-7B
j
Would you like to share the
You can remove the line 
chat_template: mistral-instruct
and try again
z
ok, trying now
j
> I am just trying to get back to where i was before the OpenLLM update. Got. Really want to make sure you can get your use case back with the new design.
z
i do really like this suite of tools. I will admit, this python stuff is a bit different to web development 😅
Where do I extend the model's maximum context length in the Chat UI?
Copy code
{
  "object": "error",
  "message": "This model's maximum context length is 4096 tokens. However, you requested 4116 tokens in the messages, Please reduce the length of the messages.",
  "type": "BadRequestError",
  "param": null,
  "code": 400
}
j
It seems a hard limit of the model itself?
z
in 
recipe.yaml
i set:
Copy code
max_tokens:
  default: 8192
  maximum: 8192
  minimum: 128
  title: Max Tokens
  type: integer
j
Ic. It should be a server side issue. The current workaround is just let UI send messages in a 4k slide windows.
z
how do I get it to use both GPUs?
j
You may add "workers=2" to service configs
z
Here is my 
recipe.yaml
Copy code
"mistral:7b-dolphin-bf16":
  project: vllm-chat
  service_config:
    name: dolphin2_9_3
    workers: 2
    traffic:
      timeout: 300
    resources:
      gpu: 2
      gpu_type: nvidia-rtx-3090
  engine_config:
    model: cognitivecomputations/dolphin-2.9.3-mistral-7B-32k
    max_model_len: 28704
    enforce_eager: true
    dtype: half
  extra_labels:
    openllm_alias: 7b
    model_name: cognitivecomputations/dolphin-2.9.3-mistral-7B-32k
  max_tokens:
  default: 28704
  maximum: 28704
  minimum: 128
  title: Max Tokens
  type: integer
I am getting out of memory errors but i can run it when 
workers: 1
Copy code
2024-08-03T00:23:41+0000 [ERROR] [entry_service:dolphin2_9_3:2] Traceback (most recent call last):
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/starlette/routing.py", line 732, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/bentoml/_internal/server/base_app.py", line 74, in lifespan
    await on_startup()
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 275, in create_instance
    self._service_instance = self.service()
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/_bentoml_sdk/service/factory.py", line 257, in __call__
    instance = self.inner()
  File "/home/bentobox/.openllm/repos/github.com/zak23/zaks-openllm-models/main/bentoml/bentos/dolphin2_9_3/7b-dolphin-bf16-e2f5/src/service.py", line 98, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(ENGINE_ARGS)
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 386, in from_engine_args
    engine = cls(
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 340, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 462, in _init_engine
    return engine_class(*args, **kwargs)
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 222, in __init__
    self.model_executor = executor_class(
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in __init__
    self._init_executor()
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 24, in _init_executor
    self.driver_worker.load_model()
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/worker/worker.py", line 121, in load_model
    self.model_runner.load_model()
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 134, in load_model
    self.model = get_model(
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
    return loader.load_model(model_config=model_config,
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 240, in load_model
    model = _initialize_model(model_config, self.load_config,
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 91, in _initialize_model
    return model_class(config=model_config.hf_config,
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 332, in __init__
    self.model = LlamaModel(config,
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 262, in __init__
    self.layers = nn.ModuleList([
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 263, in <listcomp>
    LlamaDecoderLayer(config=config,
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 200, in __init__
    self.mlp = LlamaMLP(
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 69, in __init__
    self.down_proj = RowParallelLinear(input_size=intermediate_size,
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 707, in __init__
    self.quant_method.create_weights(
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 79, in create_weights
    weight = Parameter(torch.empty(sum(output_partition_sizes),
  File "/home/bentobox/.openllm/venv/1410704492953778601/lib/python3.10/site-packages/torch/utils/_device.py", line 78, in __torch_function__
    return func(*args, **kwargs)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU
j
Oh. We can also add 
tensor_parallel_size: 2
to engine configs
z
So it would be something like
Copy code
"mistral:7b-dolphin-bf16":
  project: vllm-chat
  service_config:
    name: dolphin2_9_3
    workers: 2
    traffic:
      timeout: 300
    resources:
      gpu: 2
      gpu_type: nvidia-rtx-3090
  engine_config:
    model: cognitivecomputations/dolphin-2.9.3-mistral-7B-32k
    max_model_len: 28704
    enforce_eager: true
    dtype: half
    tensor_parallel_size: 2
  extra_labels:
    openllm_alias: 7b
    model_name: cognitivecomputations/dolphin-2.9.3-mistral-7B-32k
  max_tokens:
  default: 28704
  maximum: 28704
  minimum: 128
  title: Max Tokens
  type: integer
Thank you for your help!!! I got it working with a 7b model, but i was hoping to run a 12b model across the 2x 3090
Copy code
bentobox@bentobox:~/zaks-openllm-models$ openllm hello
  Detected Platform: linux
  Detected Accelerators:
   - NVIDIA GeForce RTX 3090 24GB
   - NVIDIA GeForce RTX 3090 24GB
? Select a model dolphin2_9_3       default  Yes
? Select a version dolphin2_9_3:12b  Yes
? Select an action 1. Serve the model locally and get a chat server
Access the Chat UI at <http://localhost:3000/chat> (or with you IP)


$ export BENTOML_HOME=/home/bentobox/.openllm/repos/github.com/zak23/zaks-openllm-models/main/bentoml
$ source /home/bentobox/.openllm/venv/142198802695865913/bin/activate
$ bentoml serve dolphin2_9_3:12b-dolphin-fp16-6649
2024-08-07T03:49:42+0000 [INFO] [cli] Starting production HTTP BentoServer from "service:VLLM" listening on <http://localhost:3000> (Press CTRL+C to quit)
/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
WARNING 08-07 03:49:47 config.py:1155] Casting torch.bfloat16 to torch.float16.
WARNING 08-07 03:49:47 config.py:427] Possibly too large swap space. 8.00 GiB out of the 15.61 GiB total CPU memory is allocated for the swap space.
2024-08-07 03:49:49,202 INFO worker.py:1781 -- Started a local Ray instance.
INFO 08-07 03:49:50 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='cognitivecomputations/dolphin-2.9.3-mistral-nemo-12b', speculative_config=None, tokenizer='cognitivecomputations/dolphin-2.9.3-mistral-nemo-12b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=20480, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=cognitivecomputations/dolphin-2.9.3-mistral-nemo-12b)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 08-07 03:49:59 utils.py:618] Found nccl from library libnccl.so.2
INFO 08-07 03:49:59 pynccl.py:65] vLLM is using nccl==2.20.5
(RayWorkerWrapper pid=3449) INFO 08-07 03:49:59 utils.py:618] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=3449) INFO 08-07 03:49:59 pynccl.py:65] vLLM is using nccl==2.20.5
INFO 08-07 03:49:59 custom_all_reduce_utils.py:179] reading GPU P2P access cache from /home/bentobox/.config/vllm/gpu_p2p_access_cache_for_0,1.json
WARNING 08-07 03:49:59 custom_all_reduce.py:167] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(RayWorkerWrapper pid=3449) INFO 08-07 03:49:59 custom_all_reduce_utils.py:179] reading GPU P2P access cache from /home/bentobox/.config/vllm/gpu_p2p_access_cache_for_0,1.json
(RayWorkerWrapper pid=3449) WARNING 08-07 03:49:59 custom_all_reduce.py:167] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
INFO 08-07 03:50:00 weight_utils.py:207] Using model weights format ['*.safetensors']
(RayWorkerWrapper pid=3449) INFO 08-07 03:50:00 weight_utils.py:207] Using model weights format ['*.safetensors']
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148] Error executing method load_model. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148] Traceback (most recent call last):
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]   File "/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 140, in execute_method
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]     return executor(*args, **kwargs)
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]   File "/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/vllm/worker/worker.py", line 121, in load_model
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]     self.model_runner.load_model()
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]   File "/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 134, in load_model
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]     self.model = get_model(
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]   File "/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]     return loader.load_model(model_config=model_config,
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]   File "/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 243, in load_model
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]     model.load_weights(
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]   File "/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 408, in load_weights
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]     weight_loader(param, loaded_weight, shard_id)
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]   File "/home/bentobox/.openllm/venv/142198802695865913/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 625, in weight_loader
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148]     loaded_weight = loaded_weight.narrow(output_dim, start_idx,
(RayWorkerWrapper pid=3449) ERROR 08-07 03:50:02 worker_base.py:148] RuntimeError: start (640) + length (640) exceeds dimension size (1024).
j
It is an engine issue. mistral-nemo has different net structure. It cannot be adapted by just config tweaking. But we are also working on it. You can wait for our updates!
z
ok, thank you! i will wait for updates 😄
2 Views
Open in Slack
PreviousNext
Powered by