Hi Team, i've just updated to the latest version of openllm and I can't longer use mistralai--Mistral-7B-Instruct-v0.2 model. Is it possible to run this model?

CC : @Jiang do you know about it

Does

openllm serve mistral:7b

work for you?

we've tried that model but it has a small context size, so it doesn't work for our use-case. Mistral-7B-Instruct-v0.2 has a 32k context size.

Hi @Boban Pavloski We are designing a feature for easily tweaking these parameters. Before that, you can work around like this:

cd ~/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/mistral/7b...

find out openllm_config.yaml, make some modification of engine configs, then run

openllm serve mistral:7b

again

Thank you for your response @Jiang , we will take a look at the configs. Thanks a lot.