https://www.bentoml.com/blog/benchmarking-llm-inference-backends Hello guys, I am trying to reproduce the tokens per seconds of this article with vllm. But my results are much lower. Which optimizations did you use to get this amount of tokens per second?

In this blog post, we try to use the default settings without any performance fine-tuning (internally we can get to better results actually) - could you check the library version, the input prompt used for testing and GPU/model used?

@Chaoyu I was wondering if you could explain something to me regarding the tokens per seconds found in your research compared to industry standard. In the first image I attached we see the tokens per second from lmdeploy having 4000 tokens per second with 100 concurrent users. This amounts to 40 tokens per second for each individual user. Now on this website: https://artificialanalysis.ai/models/llama-3-1-instruct-8b/providers We can see the output tokens per second for serverless API providers which undoubtedly make use of a large amount of concurrent users to efficiently utilize their hardware. Here we see that the slowest API (Amazon) provides 93 tokens per second with their API, which is more than 2 times faster than the best benchmark you have tested with 100 concurrent users. Do you have an idea where this difference come from? Is it because they are actually not using concurrent users? Or is it because they use tricks that are not used with your benchmark? What do you think?

The token per second per request drops as the number of concurrent requests goes up. If you look at the 10 concurrent users chart, it’s about 80-90 token per second. And yes there are many ways to improve the performance further, the benchmark we publish focuses on the default settings without any tuning. This means you if you want to ensure faster token per second, you can limit the concurrency per replica - for example, in BentoML, you can set the target concurrency parameter for your service ensuring the maximum amount of concurrent request for each replica. This can also be used for autoscaling in BentoCloud. There’re also techniques to tradeoff throughput for latency, which can be applicable for low batch scenarios. For endpoint providers, over provisioning GPUs can get you those numbers shown on the benchmark but in my experience, they fluctuates a lot depending on how many users are calling their APIs.

Thank you. My observation was for sure not a flack regarding your benchmark, it is just trying to understand what these serverless APIs are doing exactly. However, these stats that are shown are the median tokens per second, meaning that it is a fairly stable measure of their performance. It would not make sense to me for these providers to utilize a low amount of concurrent users, as otherwise the price they are offering does not seem to be sustainable. This makes me think they use different techniques. What is your opinion on this?

If you combine the number from the blog post with the most basic inference setup and medium batch size (e.g. 2500 token/s total), find the cheapest A100 on any GPU cloud(~$1.29/hr) assuming 80% overall utilization.. you are already looking at about 18 cents/million token. Like I mentioned there is still room for inference optimizations, plus they can reserve GPUs at a much lower pricing than normal on-demand GPUs - that’s where they have the extra margin.

Based on your article I am trying to do a price comparison analysis between different GPUs. Is there any information on the purchase cost price range of A100s when ordered in bulk for example?