Hey, I've been using OpenLLM as the primary way to...
# ask-for-helpt
Tristan
05/29/2024, 6:50 AMHey, I've been using OpenLLM as the primary way to submit prompts to a local LLM running on an A100.
Are there any tips to squeeze out performance for concurrent inference requests? Currently it looks like inference happens sequentially - I'm assuming that openllm doesn't handle batching automatically, and I'll need to wrap it in a BentoML and use adaptive batching or something?
c
Chaoyu
05/29/2024, 7:52 AMHI @Tristan - what version of OpenLLM are you currently using?
Chaoyu
05/29/2024, 7:53 AMthe latest version should have continuous batching by default
t
Tristan
05/29/2024, 11:16 PM0.4.44
Will upgrade to latest - cheers
👍 1
Tristan
05/30/2024, 4:49 AMIs there any additional infrastructure or configuration I can do to boost inference for a large number of requests?
Slowdown appears to occur at around 40 concurrent requests - which is sufficient - but if I can push it as much as I can that'd be groovy
c
Chaoyu
06/03/2024, 10:14 PMBentoCloud is probably what you’re looking for if you’re looking to support a production facing use case