While using Livekit Inferences, we are getting very unstable LLM TTFT which is ranging from 300ms to >5s. This is happening with over 20% of the calls right now. The Agent is deployed in LiveKit Cloud. Is there a way to mitigate this? Can we have something which is stable. The two models that I've used is gpt-4.1-mini(openai) and the other is kimi-k2 (baseten). Any help regarding this is much appreciated.

Can you isolate it to find the root cause? I assume it's not the model if it's happening with 2 different models. I suggest comparing against https://github.com/livekit-examples/agent-starter-python/blob/main/src/agent.py as a baseline, then introducing your custom agent logic to find the source (prompt length, turn detection). You say "While using Livekit inference", but I assume this doesn't go away if you use the plugin.