BentoML

Hey guys!

Great article:

<https://www.bentoml.com/blog/benchmarking-llm-inference-backends>

I was wondering, why do we test with 10, 50 and 100 users? Isn't it also interesting to test single queries?

I can imagine that for many users of LLMs it is not feasible to fill up a batch of more than 1 prompt.

input queries are continuously batched, which means unlike dynamic batch, there’s no need to wait to form a batch in LLM serving

low batch comparisons would be more interesting when speculative encoding is enabled. other wise I’d imagine the result to be similar to the 10 user scenario

Hmm I see. So how does continuous batching work exactly? Don't you need to wait to get 10 user queries before you can produce 10 outputs?

Continuous batching happens in iteration level scheduling 

So new input prompt comes in will be immediately add to the batch 

Finished generation will be removed from the batch 

Gotcha. But isnt the first instance in the batch at a different process in the forward pass for example?

You are probably referring to the prefill stage? Prefill stage can’t be batched, batching starts at the decoding stage which progressively generates new tokens 

Right right. Thats what I mean. Each query needs its own prefill stage right? So a batch of 10 queries needs to either be processed in parallel or it needs to wait until a previous one is finished. Correct?