Big Deal for Daft: As mentioned in #C08424P0A8K, we cut the 0.6.9 release today. This was a quick update to get @Kevin Wang’s new experimental vLLM provider merged for the Ray Summit: > This new vLLM Prefix Caching provider is able to accomplish this by combining the power of the vLLM serving engine with Daft’s distributed execution Flotilla to do two things: > • Dynamic Prefix Bucketing - improving LLM cache usage by bucketing and routing by prompt prefix. > • Streaming-Based Continuous Batching - Pipeline data processing with LLM inference to fully utilize GPUs. > Combined, these two strategies yield significant performance improvements and cost savings that scale to massive workloads. We observe that on a cluster of 128 GPUs (Nvidia L4), we are able to complete an inference workload of 200k prompts totaling 128 million tokens up to 50.7% faster. If you are interested in learning more, I highly recommend you read Kevin's blog post. Its pretty awesome: https://www.daft.ai/blog/cutting-llm-batch-inference-time-in-half-dynamic-prefix-bucketing-at-scale