it’s working fine on my size

And I am using an env which has only openllm installed

Pip install openllm. Is the only cmd done into this env

And I deleted the huggingface cached model and .openllm folder and initiated everything from scratch but still same error

Then at the end it loads cuda graphs and keeps saying application startup failed

Then goes into this loop re loads cuda graphs and application startup failed until I terminate it

and

openllm --version

openllm, 0.6.17

this is the exact error: 2025-01-27T082732+0000 [ERROR] [entry_servicellama3.11] Traceback (most recent call last): File "/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/starlette/routing.py", line 693, in lifespan async with self.lifespan_context(app) as maybe_state: File "/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/contextlib.py", line 181, in aenter return await self.gen.__anext__() File "/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/bentoml/_internal/server/base_app.py", line 76, in lifespan await ret File "/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/_bentoml_impl/server/app.py", line 277, in create_instance self._service_instance = self.service() File "/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/_bentoml_sdk/service/factory.py", line 302, in call instance = self.inner() File "/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/llama3.1/8b-instruct-awq-4bit-b149/src/service.py", line 118, in init vllm_api_server.init_app_state(self.engine, model_config, openai_api_app.state, args) File "/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py", line 651, in init_app_state chat_template_content_format=args.chat_template_content_format, AttributeError: 'Namespace' object has no attribute 'chat_template_content_format' 2025-01-27T082732+0000 [ERROR] [entry_servicellama3.11] Application startup failed. Exiting. 2025-01-27T082732+0000 [ERROR] [entry_servicellama3.11] Application startup failed. Exiting.

@Chaoyu I managed to fix it but that was solved manually as I went to the files I have of the model and I needed to modify the args being passed. so I think sth is wrong with some compatibility of versions during download maybe. I tried also to check the latest updates on the models files but there is nth new, however I am curious why that happened maybe there was a layer in the middle handling those args but anyways those are the two args I needed to add manually to the model files: args.chat_template_content_format = "auto" args.enable_prompt_tokens_details = False please let me know when you check it and how this can be solved automatically. thanks for help