Hi -- Just learning up a bit on BentoML so please bear with the newbie question -- In this picture -- the BentoCloud "Compute Orchestration Engine" -- do we have any details of docs? Is it underlying using a K8s orchestrated Infrastructure? Context: I am working on a K8s Gateway project for LLM Inference, and want to understand if the LLM serving optimization can be ported to this BentoCloud compute platform for improved performance & efficiency

Hi @Vaibhav Katkade - would love to hear more about your LLM serving optimization. Most likely it should be possible to port over and leverage BentoCloud (e.g. we have straight forward support for all common LLM inference runtimes https://docs.bentoml.com/en/latest/examples/overview.html#llms) Yes the orchestration engine is built on top of K8s, with some enhancements for fast coldstarts (e.g. our own container runtime, image format, model streaming loading) and efficient scaling - plus the capability to schedule across clouds and regions and the security features around the infra.

We have developed extensions to K8s Gateway API allowing for optimized LLM serving based. See: https://gateway-api-inference-extension.sigs.k8s.io/#key-features Would love to have an exploratory discussion on whether these could be a fit for BentoCloud ; or even develop some solution deployment guide with DIY BentoML