INFO 02-05 161309 model_runner.py:1099] Loading model weights took 14.9888 GB INFO 02-05 161309 worker.py:241] Memory profiling takes 0.65 seconds INFO 02-05 161309 worker.py:241] the current vLLM instance can use total_gpu_memory (44.53GiB) x gpu_memory_utilization (0.90) = 40.07GiB INFO 02-05 161309 worker.py:241] model weights take 14.99GiB; non_torch_memory takes 0.11GiB; PyTorch activation peak memory takes 1.19GiB; the rest of the memory reserved for KV Cache is 23.79GiB. INFO 02-05 161310 gpu_executor.py:76] # GPU blocks: 12179, # CPU blocks: 2048 INFO 02-05 161310 gpu_executor.py:80] Maximum concurrency for 2048 tokens per request: 95.15x INFO 02-05 161311 model_runner.py:1415] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing

gpu_memory_utilization

or switching to eager mode. You can also reduce the

max_num_seqs

as needed to decrease memory usage. Capturing CUDA graph shapes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [0015<0000, 2.31it/s] INFO 02-05 161327 model_runner.py:1535] Graph capturing finished in 15 secs, took 0.91 GiB INFO 02-05 161327 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 17.85 seconds 2025-02-05T161327-0500 [INFO] [entry_servicellama3.11] VLLM service initialized with model: meta-llama/Meta-Llama-3.1-8B-Instruct INFO 02-05 161327 api_server.py:640] Using supplied chat template: INFO 02-05 161327 api_server.py:640] None 2025-02-05T161327-0500 [ERROR] [entry_servicellama3.11] Initializing service error Traceback (most recent call last): File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/_bentoml_sdk/service/factory.py", line 302, in call instance = self.inner() File "/ltmp/home/vllm/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/llama3.1/8b-instruct-fp16-cbdd/src/service.py", line 118, in init vllm_api_server.init_app_state(self.engine, model_config, openai_api_app.state, args) File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py", line 651, in init_app_state chat_template_content_format=args.chat_template_content_format, AttributeError: 'Namespace' object has no attribute 'chat_template_content_format' 2025-02-05T161327-0500 [ERROR] [entry_servicellama3.11] Traceback (most recent call last): File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/starlette/routing.py", line 693, in lifespan async with self.lifespan_context(app) as maybe_state: File "/ltmp/home/vllm/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/contextlib.py", line 181, in aenter return await self.gen.__anext__() File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/bentoml/_internal/server/base_app.py", line 76, in lifespan await ret File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/_bentoml_impl/server/app.py", line 277, in create_instance self._service_instance = self.service() File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/_bentoml_sdk/service/factory.py", line 302, in call instance = self.inner() File "/ltmp/home/vllm/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/llama3.1/8b-instruct-fp16-cbdd/src/service.py", line 118, in init vllm_api_server.init_app_state(self.engine, model_config, openai_api_app.state, args) File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py", line 651, in init_app_state chat_template_content_format=args.chat_template_content_format, AttributeError: 'Namespace' object has no attribute 'chat_template_content_format' 2025-02-05T161327-0500 [ERROR] [entry_servicellama3.11] Traceback (most recent call last): File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/starlette/routing.py", line 693, in lifespan async with self.lifespan_context(app) as maybe_state: File "/ltmp/home/vllm/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/contextlib.py", line 181, in aenter return await self.gen.__anext__() File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/bentoml/_internal/server/base_app.py", line 76, in lifespan await ret File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/_bentoml_impl/server/app.py", line 277, in create_instance self._service_instance = self.service() File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/_bentoml_sdk/service/factory.py", line 302, in call instance = self.inner() File "/ltmp/home/vllm/.openllm/repos/github.com/bentoml/openllm-models/main/bentoml/bentos/llama3.1/8b-instruct-fp16-cbdd/src/service.py", line 118, in init vllm_api_server.init_app_state(self.engine, model_config, openai_api_app.state, args) File "/ltmp/home/vllm/.openllm/venv/1004407653884267918/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py", line 651, in init_app_state chat_template_content_format=args.chat_template_content_format, AttributeError: 'Namespace' object has no attribute 'chat_template_content_format' 2025-02-05T161327-0500 [ERROR] [entry_servicellama3.11] Application startup failed. Exiting. 2025-02-05T161327-0500 [ERROR] [entry_servicellama3.11] Application startup failed. Exiting.