Congrats on the launch team! I had a few questions up front as I'm thinking through whether/when we can utilize LiveKit Inference. 1. Classic latency consideration is to keep STT, LLM and TTS as close to our agent service as possible. What is the mental model we should have now using the Inference service? From the perspective of a self-hosted agent + LiveKit Cloud, how do we proactively ensure that we've done our part to optimize for latency? Under the Global co-location section of the blog entry announcing Inference, it seems to suggest that latency benefits only kick in when deploying agents to LiveKit Cloud... is that the case? 2. Currently we have to manage through provider outages via FallbackAdapters. Is it the intention that Inference would obviate the need for a FallbackAdapter altogether, or is it possible/recommended to configure LiveKit Inference with a fallback to provider public inference endpoints? 3. I assume LiveKit Inference will not be able to handle features like custom cloned TTS voices via its provisioned inference capacity, only the standard voices. Can you confirm whether this is the case?

Thank you! We're excited about LiveKit Inference and eager for feedback! 1. I'll get back to you about latency considerations when using LiveKit Inference with self-hosted agents. I want to make sure I'm giving you an accurate answer and there are some aspects I need to confirm with our team 2. Our goal is to handle fallbacks in LiveKit Inference, but we haven't built that yet. In the meantime, I agree with your suggestion of configuring your agent to use the FallbackAdapter and have the FallbackAdapter point to a secondary provider served through LK Inference. 3. Our first version of LK Inference does not support cloned voices. It's on our roadmap to support it sometime in the future. Thank you for your thoughtful questions.

Thanks Tom. So it really seems that to reap the benefit of Inference the agent would have to be deployed onto LiveKit Cloud, correct? Otherwise, for a self-hosted agent setup, LiveKit Inference introduces an additional network hop between the agent and the dedicated provisioned model endpoints, and we're just hoping that the Inference<>Model hop is fast enough to offset the additional hop introduced by Inference. Just want to understand the tradeoffs fully so we can make an informed decision.

Yes, I think you have it right, Mike. Given that it will depend on your specific set up, people who are not using LK hosted agents should measure whether they're getting a latency improvement through the gateway. Note that we're working on monitoring provider latency and routing around outages and slow downs, as we described some in our blog post. When we release that, the inference service will provider even more latency improvements.