I searched on this Slack, and this thread from @Zak Ozbourne looks relevant. I see the recipe for the default recipe.yaml. Is it possible to modify these values for the default models we currently have deployed? I think I understand how we could do it by deploying a custom model.

cc @Jiang could you help?

@Richard Durkee I have a bit of a guide on what to do. Its not written for other people, just myself. Would you like it?

@Zak Ozbourne Yes, I would love that, thank you! I am not particularly educated on LLMs (I'm mostly involved in systems research), so any help would be appreciated.

Ive gone over my notes, added some help text and included a model build script ### How to install and setup openllm with custom models

sudo apt install python3-pip

pip install -U bentoml

pip install -U openllm

pip install vllm

this one takes a while

pip install --upgrade jsonschema

create a fork of

<https://github.com/bentoml/openllm-models>

(https://github.com/bentoml/openllm-models/fork) Name it

custom-openllm-models

Clone to your machine

git clone <https://github.com/{git_username}/custom-openllm-models.git>

------------------------------ To add your own models, navigate to

~/custom-openllm-models/src

and edit

recipe.yaml

Here is an example, add this to the end of the

recipe.yaml

You may need to edit these depending on your model type and size.

"mistral:7b-dolphin-bf16":
  project: vllm-chat
  service_config:
    name: dolphin2_9_3
    workers: 1
    traffic:
      timeout: 300
    resources:
      gpu: 1
      gpu_type: nvidia-rtx-3090
  engine_config:
    model: cognitivecomputations/dolphin-2.9.3-mistral-7B-32k
    max_model_len: 28704
    enforce_eager: true
    dtype: half
    tensor_parallel_size: 1
  extra_labels:
    openllm_alias: 7b
    model_name: cognitivecomputations/dolphin-2.9.3-mistral-7B-32k
  max_tokens:
  default: 28704
  maximum: 28704
  minimum: 128
  title: Max Tokens
  type: integer

--------------------

cd ~/custom-openllm-models/src/vllm-chat

pip install -r requirements.txt -U

this one also takes a long time

cd ~/custom-openllm-models/src

BENTOML_HOME=/home/bentobox/custom-openllm-models/bentoml/ python3 make.py "mistral:7b-dolphin-bf16"

Use the Title of the custom model from

recipe.yaml

"mistral:7b-dolphin-bf16"

nano ~/.bashrc

echo 'export PATH=$PATH:/home/bentobox/.local/bin' >> ~/.bashrc

source ~/.bashrc

Remove the default openLLM repo and add your new custom one:

openllm repo remove default

`openllm repo add default https://github.com/{username}/custom-openllm-models

openllm repo update

Run OpenLLM

openllm hello

You should see a list of models and options on how to run them. the first run will take time to start as it has to download the model files ---- ## Here is a bash file i made to streamline the building process

bash
#!/bin/bash

# Prompt for the model name
read -p "Enter the model name (e.g., mistral:7b-dolphin-bf16): " MODEL_NAME

# Navigate to the specified directory
cd ~/custom-openllm-models || { echo "Failed to navigate to ~/custom-openllm-models"; exit 1; }

# Perform git pull
git pull || { echo "Git pull failed"; exit 1; }

# Navigate to src directory
cd src || { echo "Failed to navigate to src"; exit 1; }

# Run the make.py script with the specified model name
BENTOML_HOME=/home/bentobox/custom-openllm-models/bentoml/ python3 make.py "$MODEL_NAME" || { echo "Python script execution failed"; exit 1; }

# Navigate back to the parent directory
cd ..

# Perform git add, commit, and push
git add . || { echo "Git add failed"; exit 1; }
git commit -m "built $MODEL_NAME" || { echo "Git commit failed"; exit 1; }
git push || { echo "Git push failed"; exit 1; }

# Update openllm repo
openllm repo update || { echo "openllm repo update failed"; exit 1; }

echo "Script executed successfully"

------- ### Setting up OpenLLM as Service

sudo nano /etc/systemd/system/openllm-dolphin.service

[Unit]
Description=OpenLLM Dolphin Service
After=network.target

[Service]
Type=simple
User=bentobox
WorkingDirectory=/home/{username}
ExecStart=/home/{username}/.local/bin/openllm serve dolphin2_9_3:7b
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload

sudo systemctl start openllm-dolphin
sudo systemctl enable openllm-dolphin

sudo systemctl status openllm-dolphin

@Zak Ozbourne Thank you for documenting your efforts, it's very appreciated. I will have some time to try this either today or Thursday, so I will get back to you with how it went sometime this week.

@Richard Durkee how did you go?