I'm getting 429's for TTS and STT on livekit inference while running locally in "console" mode. my concurrent STT/TTS count is at 2, which is less than the 5 max on the Build plan. What could the issue be?

Thanks for reaching out, Ashwin. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I've been retrying every ~15 minutes for an hour now, still 429's. my agent config is: # Set up a voice AI pipeline using ElevenLabs, Google, and the LiveKit turn detector session = AgentSession( turn_detection="stt", stt=inference.STT(model="assemblyai/universal-streaming"), llm=inference.LLM(model="google/gemini-2.5-flash"), tts=inference.TTS( model="inworld/inworld-tts-1.5-mini", voice="Sarah", language="en" ), vad=ctx.proc.userdata["vad"], preemptive_generation=True, ) my logs look like: 145534.888 ERROR… asyncio Task exception was never retrieved future: <Task finished name='Task-19' coro=<ConnectionPool.prewarm.<locals>._prewarm_impl() done, defined at /Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/livekit/agents/utils/connection_pool.py:169> exception=APIStatusError('Invalid response status (429 Too Many Requests)', status_code=429, request_id=None, body=None, retryable=False)> Traceback (most recent call last): File "/Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/livekit/agents/inference/tts.py", line 412, in _connect_ws ws = await asyncio.wait_for( ^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... ) ^ File "/Users/ashwin/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/asyncio/tasks.py", line 507, in wait_for return await fut ^^^^^^^^^ File "/Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 1093, in _ws_connect raise WSServerHandshakeError( ...<5 lines>... ) aiohttp.client_exceptions.WSServerHandshakeError: 429, message='Invalid response status', url='wss://agent-gateway.livekit.cloud/v1/tts?model=inworld/inworld-tts-1.5-mini' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/livekit/agents/utils/connection_pool.py", line 172, in _prewarm_impl conn = await self._connect(timeout=self._connect_timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/livekit/agents/utils/connection_pool.py", line 63, in _connect connection = await self._connect_cb(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/livekit/agents/inference/tts.py", line 417, in _connect_ws raise create_api_error_from_http(e.message, status=e.status) from e livekit.agents._exceptions.APIStatusError: Invalid response status (429 Too Many Requests) 145534.918 WARNI… livekit.agents failed to recognize speech: Invalid response status (429 Too Many Requests), retrying in 0.1s {"stt": "livekit.agents.inference.stt.STT", "attempt": 0, "streamed": true, "room": "console"} 145535.013 WARNI… livekit.agents failed to synthesize speech: Invalid response status (429 Too Many Requests), retrying in 0.1s {"tts": "livekit.agents.inference.tts.TTS", "attempt": 1, "streamed": true, "room": "console"} 145535.063 WARNI… livekit.agents failed to recognize speech: Invalid response status (429 Too Many Requests), retrying in 2.0s {"stt": "livekit.agents.inference.stt.STT", "attempt": 1, "streamed": true, "room": "console"} 145535.143 WARNI… livekit.agents failed to synthesize speech: Invalid response status (429 Too Many Requests), retrying in 2.0s {"tts": "livekit.agents.inference.tts.TTS", "attempt": 2, "streamed": true, "room": "console"} 145537.150 WARNI… livekit.agents failed to recognize speech: Invalid response status (429 Too Many Requests), retrying in 2.0s {"stt": "livekit.agents.inference.stt.STT", "attempt": 2, "streamed": true, "room": "console"} 145537.172 WARNI… livekit.agents failed to synthesize speech: Invalid response status (429 Too Many Requests), retrying in 2.0s {"tts": "livekit.agents.inference.tts.TTS", "attempt": 3, "streamed": true, "room": "console"} 145537.546 INFO … livekit.agents shutting down worker {"id": "unregistered"} 145539.246 ERROR… livekit.agents Error in _stt_task {"room": "console"} Traceback (most recent call last): File "/Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/livekit/agents/inference/stt.py", line 599, in _connect_ws ws = await asyncio.wait_for( ^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... ) ^ File "/Users/ashwin/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/asyncio/tasks.py", line 507, in wait_for return await fut ^^^^^^^^^ File "/Users/ashwin/Documents/pestphone/agents/demo/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 1093, in _ws_connect raise WSServerHandshakeError( ...<5 lines>... ) aiohttp.client_exceptions.WSServerHandshakeError: 429, message='Invalid response status', url='wss://agent-gateway.livekit.cloud/v1/stt?model=assemblyai/universal-streaming'

Thanks — I see you’ve been getting repeated 429s for both TTS and STT while running in console mode, and you’ve already been retrying for an hour. You included your agent config and full traceback showing websocket handshake 429s to the LiveKit agent gateway for these models. Let’s walk through the most likely causes and a focused set of next steps to unblock you. Why this is happening • The 429s are coming from the agent gateway when opening websocket connections to the provider-backed TTS and STT endpoints (wss://agent-gateway.livekit.cloud/v1/tts and /v1/stt). This indicates throttling at the provider or gateway layer, not necessarily the LiveKit dashboard’s aggregate concurrency count. • Model/provider-level limits often differ from the Build plan’s 5-concurrent limit. For example, websocket STT streams or certain TTS models may have tighter per-model or per-provider concurrent connection limits or short-term RPS/token burst limits. • Your logs show many quick retries (0.1s, 2.0s) which can keep hitting provider throttling windows; providers sometimes return 429 until their internal rate-limited window resets. Immediate actions to try 1. Pause aggressive retries and use longer exponential backoff ◦ Increase retry delays and backoff caps. For example: start at 500 ms, then 1s, 2s, 4s, 8s, and stop after ~4 retries. This reduces repeated handshake attempts that prolong throttling. 1. Reduce concurrent websocket prewarm attempts ◦ The stacktrace shows the ConnectionPool prewarm failing. Disable or lower prewarm concurrency so only 1 connection attempts at a time while diagnosing. 1. Test single model at a time ◦ Run only STT (remove TTS) and see if websockets connect. Then test only TTS. This isolates whether one provider/model is being throttled. 1. Slow your request/stream rate in console mode ◦ Throttle your test input so you don’t open/close websockets rapidly or send many small requests back-to-back. 1. Capture the response headers/body if present ◦ If any 429 responses include a Retry-After header or provider error body, that value helps determine how long to wait. Your current log shows body=None, but check full HTTP/ws response if available. Log and config items that will help me narrow this further • Which exact providers back these models? From your config: STT = assemblyai/universal-streaming and TTS = inworld/inworld-tts-1.5-mini. Can you confirm those map to AssemblyAI and Inworld providers? (That will let me point to provider-specific limits.) • How many prewarm / pool connections does your AgentSession/connection pool create by default? If you can paste the connection pool settings or confirm if prewarm is on, that helps. • Any Retry-After header values or full error bodies from the gateway responses (if different from body=None). • Whether this happens only in console mode or also when running on a hosted environment. What I can help with next • If you confirm the providers (AssemblyAI and Inworld), I can outline typical concurrency and websocket limits to watch for and suggest concrete backoff timings. • If you paste any additional gateway error response text or the connection pool settings, I’ll recommend a small config change you can make locally to avoid repeated 429s. If you prefer, start by disabling prewarm and increasing the retry backoff (500ms -> 1s -> 2s -> 4s) and rerun a single-model test; paste the first few log lines if it still fails and I’ll take the next step with provider-specific suggestions.

I'm using inworld/assemblyai on livekit inference.

Thanks — this is helpful. You’ve confirmed you’re using Inworld (TTS) and AssemblyAI (STT) on LiveKit Inference, and your logs show websocket handshake 429s from the agent gateway for those models. I want to get you unblocked quickly, so I’ll focus on what to try next and a few targeted questions that will let me recommend concrete, low-friction fixes. What to try right now (fast checks) • Stop aggressive prewarm/retries: Disable or reduce connection prewarm and increase retry backoff. Set retries like 0.5s → 1s → 2s → 4s with jitter and cap at ~3–4 attempts. • Test one provider at a time: Run only AssemblyAI STT, confirm websocket connects; then run only Inworld TTS. This isolates which provider is throttling. • Reduce request/stream frequency in console mode: Throttle input so you’re not opening/closing websockets or sending many short requests rapidly. • Capture any Retry-After header: If a 429 returns a Retry-After header, that value tells you how long to wait before retrying. Why this is likely happening • Inworld and AssemblyAI each enforce their own concurrency/RPS limits. LiveKit Inference cannot override those provider-level limits, so you can receive 429s from the gateway even if LiveKit’s dashboard shows available aggregate capacity. • Your logs show many short retries (0.1s, 2.0s), which can keep hitting the provider throttling window and prolong the 429s. Targeted info I need to give a precise fix Please provide any of the following you can share — even a couple will help me pinpoint the next steps: 1. LiveKit Cloud plan (Build / Ship / Scale). 2. The connection pool / prewarm settings in your agent (are you using default prewarm? pool size?). 3. Whether you see any Retry-After header or any non-empty response body with the 429 (your current logs show body=None, but raw handshake headers would help). 4. Counts at the moment of failure: ◦ number of agent sessions / rooms active ◦ concurrent STT websocket connections ◦ concurrent TTS websocket connections 1. Whether this happens only in local console mode or also on deployed hosts. 2. If you have direct Inworld or AssemblyAI accounts, whether those dashboards show any quota/rate-limit hits. If you want to try one immediate quick experiment: disable prewarm, run a single STT websocket session to AssemblyAI, and paste the first 20 log lines. If it still 429s, include the raw websocket handshake headers (if available) — that will confirm provider throttling and let me recommend exact backoff timing or connection-pooling tweaks for your setup. I can then provide: • Provider-specific concurrency and retry guidance for Inworld and AssemblyAI. • Exact connection-pool / prewarm config suggestions to avoid stampeding and persistent 429s. Sources: Introducing LiveKit Inference: A unified model interface for voice AI | How much can LiveKit scale? | LiveKit | examples/survey | livekit/livekit_composite | DeepWiki | server-sdk-go | livekit/livekit_composite | DeepWiki