BentoML

Where can I override these setting for the openllm chat ui: Service Configuration
This model was configured with the following settings:
```{
  "enable_auto_tool_choice": true,
  "max_model_len": 3192,
  "tensor_parallel_size": 1,
  "tool_call_parser": "llama3_json"
}```
I want to change the max_model_len to a higher number. I don't know where the configuration file is located.

are you using the ui from `openllm serve` ?

you might have to update the bentos itselves. it is under `~/.openllm/repos/github.com/bentoml/openllm-models/main`

yes, i am using the chat ui from 'openllm serve llama3.1:8b-instruct-239c  --port 37090'. i tried editing max_model_len in the /var/local/home/openllm/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/llama3.1/8b-instruct-239c/src/service.py and set it to " max_model_len: int = 13192".  It doesn't take. is there an upper limit you can't exceed?

does the max_model_len setting set the max token for the chat ui?