Hi guys, I'm new to Livekit, and I'm getting the following error: "aiohttp.client_exceptions.WSServerHandshakeError: 429, message='Invalid response status', url='wss://agent-gateway.livekit.cloud/v1/tts'" However, when checking the consumption limits, it doesn't seem like I've exceeded any limits.

Thanks for reaching out, Gabriel. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

A new websocket is not opened for each text message, but rather one per TTS request; multiple fragments are sent over that same socket. However, when there are concurrent requests, each will get its own websocket because the pool does not share a connection that is already in use.

Thanks — that clarification helps. You're creating one WebSocket per concurrent TTS request (pool not sharing sockets), which can trigger gateway rate limits even though per‑minute quotas look fine. Here are focused, practical options you can apply now. What to try first • Limit concurrency at the client/worker level. Cap active TTS sockets (for example, max 3 concurrent) and queue additional requests. This prevents bursts of simultaneous handshakes that cause 429s. • Use a short request queue with worker concurrency and backpressure rather than allowing unbounded parallel requests. Backoff & retry • On handshake failures (429), retry with exponential backoff and jitter, and a small retry cap: ◦ Example pattern: wait (2**attempt) * 100 ms + random(0–100 ms), attempts ≤ 3. • Ensure failed attempts don’t immediately requeue unlimited retries—count failed attempts per request. Connection reuse improvements • If possible, allow the pool to reuse an idle TTS socket for a new request even if it was previously used by another job (i.e., return sockets to pool immediately after the TTS stream completes). • If requests require exclusive use of a socket while streaming, consider keeping a small pool of long‑lived sockets and rotate requests through that pool. Monitoring & tuning • Add instrumentation: track handshake 429 counts, current concurrent TTS sockets, and requests/sec. Tune concurrency down until 429s disappear, then raise gradually. • If you see 429s spike only at certain times, throttle only during spikes. Longer-term options • Upgrade to Scale for higher rate limits. • Use a TTS plugin with your own provider key (e.g., ElevenLabs) to offload TTS from the agent gateway. If you want, share: • the pool size you currently use, • sample handshake error timestamps or rate of concurrent open attempts, and I’ll recommend a specific concurrency cap and a retry/backoff implementation you can drop into your code.

This is the agent's code.

Thanks — I can review the agent code and point out exact changes. Please paste the TTS‑related code that manages the WebSocket pool, connection creation/checkout, and where requests are queued/retried (the functions or methods that open the socket, send fragments, and release or close the socket). If the snippet is long, send the relevant parts for: • pool creation/configuration (pool size, idle timeout) • handshake/open connection logic • per‑request flow (checkout, send fragments, wait for completion, release) • retry/backoff handling Also tell me: • current pool size, • typical concurrent requests you see when 429s occur, • SDK/language version you’re using. I’ll review and give concrete code edits and exact retry/concurrency values to stop the 429s.

That's all my code related to the agent; the rest are LiveKit's own files. I don't handle connections manually; I'm using the default settings.

Thanks — that helps a lot. Since you’re using the SDK’s default pool behavior (one websocket per concurrent TTS request) and not managing connections yourself, the 429 is almost certainly caused by short‑term rate or concurrency limits (bursty handshakes) rather than overall quota exhaustion. Here are specific, low‑effort steps you can apply right now and what I can do next to help. What to try immediately • Reduce effective concurrency client-side without changing SDK internals: ◦ Add a simple semaphore or worker queue around TTS calls so only N requests may start a websocket at once. Start with N = 3 and raise if stable. • Add retry with exponential backoff on handshake 429s: ◦ Pattern: attempts ≤ 3, wait = (2*_attempt)_100 ms + random(0–100 ms). • Shorten aggressive retries elsewhere: ensure failed TTS handshakes aren’t immediately requeued repeatedly. Why this helps • The gateway enforces per-second/minute and concurrent connection protections; capping simultaneous handshake attempts prevents bursts that trigger 429s even when dashboard quotas look fine. If you want me to give exact code • Tell me which SDK/language and version you’re using (JS, Python, Go, etc.). I’ll provide a minimal snippet to wrap TTS calls with: ◦ a concurrency semaphore/queue, and ◦ retry + exponential backoff logic you can drop in without changing LiveKit internals. Other options if this persists • Increase concurrency by upgrading to Scale (higher limits). • Offload TTS to a plugin using your own TTS provider key (e.g., ElevenLabs) to avoid the agent gateway rate limits. Which SDK/language are you using so I can provide the concrete code snippet? Sources: Quotas and limits | LiveKit docs

python

Thanks — since you’re using Python, here’s a compact, ready-to-drop solution you can use without changing SDK internals. It: • caps concurrent TTS websocket handshakes to avoid bursts that trigger 429s, and • retries handshake failures (429) with exponential backoff + jitter. Paste this into the part of your app that dispatches/starts agent TTS sessions.

import asyncio
import random
from contextlib import asynccontextmanager
from livekit.agents import AgentSession  # adjust import to your SDK layout

# Tune this to <= your plan's concurrent TTS limit (start with 3)
MAX_CONCURRENT_TTS = 3
_tts_semaphore = asyncio.Semaphore(MAX_CONCURRENT_TTS)

async def _backoff_sleep(attempt: int):
    base_ms = (2 ** attempt) * 100
    jitter = random.uniform(0, 100)
    await asyncio.sleep((base_ms + jitter) / 1000.0)

@asynccontextmanager
async def tts_session_with_limits(*session_args, **session_kwargs):
    """
    Acquire semaphore, attempt to create/start a TTS AgentSession with retries on 429.
    Yields the started session (caller is responsible for using and closing it).
    """
    await _tts_semaphore.acquire()
    try:
        max_attempts = 3
        for attempt in range(max_attempts):
            try:
                session = AgentSession(*session_args, **session_kwargs)
                # Replace `start(...)` with your actual session start call
                await session.start(...)   # e.g., await session.start(agent=agent, room=room)
                yield session
                return
            except Exception as e:
                # Inspect the exception to detect 429 handshake error.
                # Adjust detection based on the actual exception type/message you see.
                msg = str(e).lower()
                if "w sserverhandshakeerror" in msg or "429" in msg or "too many requests" in msg:
                    if attempt < max_attempts - 1:
                        await _backoff_sleep(attempt)
                        continue
                # Non-retryable or out of attempts -> re-raise
                raise
    finally:
        try:
            _tts_semaphore.release()
        except Exception:
            pass

Example usage:

async def handle_tts_request(...):
    async with tts_session_with_limits(tts="cartesia/sonic-3") as session:
        try:
            # send fragments over the session websocket
            await session.send_fragment(...)   # replace with actual API calls
            await session.finish()             # or close/complete pattern
        finally:
            await session.close()

Notes and tuning • Set MAX_CONCURRENT_TTS to 3 (or lower) to start; your plan’s visible concurrent TTS limit is 5, but leaving headroom prevents handshake bursts. Raise slowly if stable. • The retry logic specifically targets 429/handshake errors; if your SDK raises a specific exception class for handshake failures, replace the text check with isinstance(...) or inspect the error fields. • Instrument metrics: count handshake-429s, current semaphore waiters, and average handshake latency to tune values. If you share the exact start/handshake calls (what you replace for

await session.start(...)

and the exception class/message you actually see), I’ll refine the snippet to match your SDK calls and error detection precisely. Sources: Quotas and limits | LiveKit docs | server-sdk-go | livekit/livekit_composite | DeepWiki

If I use this configuration, the error disappears. (If I use this configuration the error disappears (stop using tts and stt to only use llm with open ai realtime api))

session = AgentSession(
        llm= openai.realtime.RealtimeModel(
            voice="marin",
            model="gpt-realtime",
            turn_detection=TurnDetection(
            type="server_vad",
            threshold=0.5,
            prefix_padding_ms=300,
            silence_duration_ms=500,
            create_response=True,
            interrupt_response=True,
        )),
        turn_detection=MultilingualModel(),
        vad=ctx.proc.userdata["vad"],
        userdata={
            "facility_id": facility_id,
            "dialed_number": dialed_number,
        },
    )

Thanks — good info. You confirmed that switching to LLM-only (OpenAI realtime) removes the 429, which means the 429 is caused by TTS/STT usage (concurrency or handshake rate limits) rather than LLM use. Helpful confirmation. Quick recommendations now that we know TTS/STT are the trigger: • Keep using your LLM-only config if you don’t need TTS/STT — that’s a valid immediate workaround. • If you need TTS/STT alongside LLM, re-enable them but wrap TTS/STT session creation with the semaphore + backoff pattern I gave earlier. Start with MAX_CONCURRENT_TTS = 3 and adjust upward only after monitoring. • Apply the same concurrency control for STT sessions if you create concurrent STT websockets (STT also counts toward concurrent agent sessions). • Instrument: count handshake 429s, current concurrent TTS/STT sockets, and requests/sec. Reduce concurrency until 429s stop, then tune. If you want, I can: • Provide a combined Python snippet that creates AgentSession with both

llm

and `tts`/`stt` while using the semaphore + retry wrapper (drop‑in for your code). • Recommend exact semaphore sizes if you share typical peak concurrent TTS+STT requests. Which would you prefer?