[Bug / Rate Limit] AgentSession fails on startup w...
# ask-ai
d
[Bug / Rate Limit] AgentSession fails on startup with 429 from inference gateway (STT + TTS) Plan: Build SDK version: livekit-agents ~= 1.2 Python: 3.12 --- What's happening: When an AgentSession starts (in console mode), the SDK's internal ConnectionPool.prewarm attempts to open WebSocket connections to the STT and TTS inference endpoints simultaneously. Both are immediately rejected with HTTP 429 before the handshake completes, causing an unrecoverable session crash. Affected endpoints: - wss://agent-gateway.livekit.cloud/v1/stt → 429 - wss://agent-gateway.livekit.cloud/v1/tts → 429 Error: aiohttp.client_exceptions.WSServerHandshakeError: 429, message='Invalid response status', url='wss://agent-gateway.livekit.cloud/v1/stt' Agent config: AgentSession( stt=inference.STT(model="deepgram/nova-3", language="en"), llm=inference.LLM(model="openai/gpt-4.1-mini"), tts=inference.TTS(model="cartesia/sonic-3"), preemptive_generation=True, ) Notes: - Billing page shows active usage (STT, TTS, LLM all consumed within the month) — not a quota exhaustion issue - No custom retry logic or multiple sessions — this happens on the very first connection attempt - The 429 response body is 495 bytes (JSON) but the SDK doesn't surface its contents Question: Is the Build plan subject to a per-second WebSocket connection rate limit on the inference gateway? If so, is there a way to configure or increase it, or disable prewarming to avoid opening connections simultaneously?
t
Thanks for reaching out, Waqas. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m sorry you hit this on startup. I’m responding to your question about whether the Build plan is subject to a WebSocket connection rate / concurrency limit on the inference gateway, and whether you can increase it or disable prewarming to avoid simultaneous connections. Short answers • The Build plan enforces LiveKit Inference concurrency limits (Build provides 5 concurrent inference connections). Simultaneous STT + TTS WebSocket opens during startup can exceed concurrency or burst limits and return HTTP 429. • There isn’t a documented SDK option to disable the SDK’s prewarm/connection behavior or to change the gateway’s per-second connection throttling. • You can request higher inference concurrency limits via the LiveKit dashboard to reduce the chance of 429s from burst connection attempts. What you can do next • Temporarily avoid simultaneous model connections by creating the AgentSession with only the models you immediately need, then lazy-init the other inference clients after startup (e.g., create/connect to TTS after STT is established). This staggers WebSocket opens and often avoids 429s. • Request an increase to your inference concurrency from the dashboard so the gateway allows more concurrent model connections for your account. • If you can share the 429 response body (the 495-byte JSON) I can help interpret whether it indicates a concurrency limit or another throttle reason. Also tell me whether this happens reproducibly on first startup or only after other activity in your account. If you’d like, I can walk through a minimal code pattern to lazily initialize STT/TT S connections in your AgentSession so they don’t open simultaneously. Which approach do you prefer? Sources: https://livekit.com/pricing.md| | Pricing | LiveKit | Build voice, video, and physical AI | LiveKit | Pricing | LiveKit | Server options | LiveKit Documentation, with interactive elements
Thanks — I see the exact 429 WSS handshake error for the STT endpoint you pasted. I’m sorry this is blocking startup. What’s happening • The gateway is returning HTTP 429 at the WebSocket handshake for
/v1/stt
(Content-Length: 495 JSON body). This indicates the gateway is rejecting the connection due to throttling/concurrency limits rather than a protocol bug in your client. • On the Build plan LiveKit enforces inference concurrency limits (Build provides 5 concurrent inference connections). Opening STT and TTS WebSockets simultaneously during AgentSession startup can count as concurrent/burst connections and trigger 429s. Immediate mitigations you can try • Stagger the connections so STT and TTS don’t open at the same time (lazy-init one after the other). This avoids the burst that triggers the 429. I can provide a short code snippet to do this in your agent if you want. • Create the AgentSession with only the inference clients you need immediately, then connect the others after startup succeeds. Longer-term options • Request a higher inference concurrency limit from the dashboard for your account so the gateway allows more concurrent model connections and reduces 429s from burst opens. Information I can use to help further • Paste the 495-byte JSON response body (it likely explains the throttle reason). • Confirm whether this 429 happens every first-run attempt even on a fresh account, and whether it reproduces if you only enable one of STT/TTS. Which would you like next — a brief code pattern to lazily initialize TTS after STT connects, or help interpreting the 429 JSON body? , with interactive elements