Hi everyone I am using bentoml for serving and I’m noticing that the inference is slow, takes about 3-4 seconds to get the output although when using openllm for serving it is much faster I am using the bentoml.api decorator, is there a certain configuration that I need to pass to speed up the inference time?

There shouldn't be a big difference since openllm is using bentoml underneath as a serving layer. could you share your bentoml code and we can inspect it together

I thought the same thing but there are differences in versions so maybe the issue is there? Here are the versions I’m using and the code is following the tutorial Openllm start: • openllm 0.4.44 • bentoml 1.1.11 • vllm 0.2.7 bentoml setve • openllm 0.4.42 • bentoml 1.2.9 • vllm 0.3.0

@bentoml.service(traffic={"timeout": 6000, "max_concurrency": 500})
class VLLM:
    def __init__(self) -> None:
        from vllm import AsyncEngineArgs, AsyncLLMEngine

        model_path = "mistral/"
        ENGINE_ARGS = AsyncEngineArgs(
            model=model_path,
            max_model_len=MAX_TOKENS,
            gpu_memory_utilization=0.9,
            max_num_seqs=1,
        )

        self.engine = AsyncLLMEngine.from_engine_args(ENGINE_ARGS)
        self.request_id = 0

    @bentoml.api(batchable=False)
    async def generate(
        self, prompt: str = "Who are you?", tokens: Optional[List[int]] = None
    ) -> AsyncGenerator[str, None]:

        SAMPLING_PARAM = SamplingParams(max_tokens=MAX_TOKENS, temperature=0)
        prompt = PROMPT_TEMPLATE.format(user_prompt=prompt)
        stream = await self.engine.add_request(
            self.request_id, prompt, SAMPLING_PARAM, prompt_token_ids=tokens
        )
        self.request_id += 1
        async for request_output in stream:
            response = request_output.outputs[0].text
        return response

@larme (shenyang) perhaps you may comment on this one

Hi Noor, is there a reproducible setup for me to test further? Are you testing the performance with a single request or concurrently multiple requests? Also may I ask why you supply

max_num_seqs=1

to the engine?

I’m sending 100 requests concurrently I was testing some other args but it’s not what is causing the issue I tested without it and the inference is still slower

what's the throughput different? Do you have a locust script to share?

The metric I’m looking at the time to first token, and the difference is 10 times the time it takes openllm Unfortunately the script is private property so I can’t share it

ok if you are testing the time to first token, then maybe because you need to enable stream. Could you use this example as a test? https://github.com/bentoml/BentoVLLM/tree/main/mistral-7b-instruct I will also do a test using mistral-7b

I’ll test that and let you know