I am getting the following error while running the livekit agent: {"message": "failed to synthesize speech, retrying in 0.1s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "worker is at full capacity, marking as unavailable", "level": "INFO", "name": "livekit.agents", "load": 0.99779, "threshold": 0.7, "timestamp": "2025-12-24T151520.64837 How to fix it?

Thanks for reaching out, shreyasraoinna. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Full log request: {"message": "received job request", "level": "INFO", "name": "livekit.agents", "job_id": "AJ_wBuBimwdEZTD", "dispatch_id": "AD_qMAY9YyHAb88", "room": "JamesCarterWed Dec 24 2025 20:45 {"message": "initializing process", "level": "INFO", "name": "livekit.agents", "pid": 32, "timestamp": "2025-12-24T151517.485892+00:00"} {"message": "starting voice assistant for participant JamesCarter", "level": "INFO", "name": "voice-assistant", "pid": 18, "job_id": "AJ_wBuBimwdEZTD", "room_id": "RM_UiFC8GQiPDtD", " {"message": "RoomInputOptions and RoomOutputOptions are deprecated, use RoomOptions instead", "level": "WARNING", "name": "livekit.agents", "pid": 18, "job_id": "AJ_wBuBimwdEZTD", "ro {"message": "failed to synthesize speech, retrying in 0.1s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "worker is at full capacity, marking as unavailable", "level": "INFO", "name": "livekit.agents", "load": 0.99779, "threshold": 0.7, "timestamp": "2025-12-24T151520.64837 {"message": "process initialized", "level": "INFO", "name": "livekit.agents", "pid": 32, "elapsed_time": 3.46, "timestamp": "2025-12-24T151520.948750+00:00"} {"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "worker is below capacity, marking as available", "level": "INFO", "name": "livekit.agents", "load": 0.5304576000000001, "threshold": 0.7, "timestamp": "2025-12-24T15152 {"message": "STT metrics", "level": "INFO", "name": "livekit.agents", "model_name": "nova-3", "model_provider": "Deepgram", "audio_duration": 4.4, "pid": 18, "job_id": "AJ_wBuBimwdEZT

Thanks — I’m responding to the full logs you pasted (repeated TTS failures, worker toggling unavailable/available, STT metrics). This gives enough to triage next steps; I need one more detail and then steps to try. Quick question: • Which TTS plugin/provider are you using (plugin name and whether it’s remote API or local model)? Also paste the full traceback text that follows the tts.py lines (the logs you showed cut off before the exception). Immediate things to try (in order) 1) Validate TTS auth & network • Confirm API key/env vars for the TTS provider are present in the agent process. • From the agent host, run a basic connectivity test to the provider (curl or the provider’s SDK quick call). If using a local model, ensure the model server is running. 2) Capture the exact TTS error • Paste the full traceback lines after the path in

livekit/agents/tts/tts.py

. That will show whether the error is auth, timeout, model load failure, or something else. 3) Reduce retry pressure • Temporarily increase the TTS retry backoff so retries don’t rapidly spike load (e.g., longer exponential backoff). This prevents retry storms that can push load over threshold. 4) Address worker capacity • The log shows load ~0.997 > threshold 0.7, then later 0.53 < 0.7 — the worker is oscillating. To fix: ◦ Short-term: raise

load_threshold

(for example to 0.9) only if your host has headroom. ◦ Best: run additional agent workers (same

agent_name

) so jobs are distributed horizontally. ◦ Consider a custom

load_fnc

if the default metric is too conservative for your environment. 5) Reduce per-job resource usage • If TTS uses local heavy models, move TTS to a dedicated machine or hosted API to avoid saturating the agent host. • Pre-warm TTS resources if supported by your plugin to avoid repeated costly initializations. 6) Add observability • Enable more detailed agent logs/metrics around CPU, memory, queue length and TTS latency to correlate the TTS failures with resource spikes. If you paste: • The full TTS traceback (exact error message) • Which TTS plugin/provider you use • Your agent config snippet showing

AgentServer

settings (if modified) I’ll give the exact next fix (auth or network steps, config snippet to change, or scaling recommendation).

This is my code: .rtc_session() async def entrypoint(ctx: JobContext): await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY) participant = await ctx.wait_for_participant() logger.info(f"starting voice assistant for participant {participant.identity}") if participant.identity: username = participant.identity else: # Adding this line so that conversations will work in sandbox frontend username = "PaulSmith" system_instructions, vector_store_ids = get_system_instructions(username) voice_instructions = "You are a voice agent. Ensure that your responses are short and concise.\n" system_instructions = voice_instructions + system_instructions vad_instance = ctx.proc.userdata.get("vad") if vad_instance is None: # fallback (should not happen if prewarm ran) vad_instance = silero.VAD.load() agent = Agent( instructions=system_instructions, vad=vad_instance, stt=deepgram.STT(), llm=openai.LLM.with_azure(azure_deployment="gpt-4o", model="gpt-4o", azure_endpoint=AZURE_OPENAI_ENDPOINT, api_key=AZURE_OPENAI_API_KEY, api_version="2024-12-01-preview"), # tts=openai.TTS(model="tts-1", voice="sage"), tts=openai.TTS.with_azure(model="tts", voice="nova", api_version="2024-02-15-preview", instructions="Speak slowly in a calm and soothing voice."), # llm=openai.LLM(model="gpt-4o"), # tts=openai.TTS(model="tts-1",voice="sage"), tools=[#query_info, email_alert_for_suicide,email_alert_to_property_manager,email_alert_to_entitlements_specialist,email_alert_to_employment_specialist], ) session = AgentSession(userdata=MySessionInfo(user_name=username, vector_store_id=vector_store_ids[0])) # log metrics as they are emitted, and total usage after session is over usage_collector = metrics.UsageCollector() @session.on("metrics_collected") def _on_metrics_collected(ev: MetricsCollectedEvent): metrics.log_metrics(ev.metrics) usage_collector.collect(ev.metrics) async def log_usage(): summary = usage_collector.get_summary() # chat_history = str(session.history.to_dict()) # write_chat_to_db(chat_history, username=username) logger.info(f"Usage: {summary}") # shutdown callbacks are triggered when the session is over ctx.add_shutdown_callback(log_usage) await session.start(agent=agent, room=ctx.room, room_input_options=RoomInputOptions( # LiveKit Cloud enhanced noise cancellation # - If self-hosting, omit this parameter # - For telephony applications, use

BVCTelephony

for best results noise_cancellation=noise_cancellation.BVC(), ), ) await session.say("Hey, how can I help you today?", allow_interruptions=True)

Thanks — I’m responding to the code and full logs you posted (TTS retries, worker oscillating between unavailable/available). This looks like two cooperating issues: Azure OpenAI TTS calls are failing at runtime, and retry activity plus TTS work is pushing the agent load near 1.0 so it temporarily marks itself unavailable. Two quick clarifying questions: • Which exact Azure TTS plugin/provider are you using? (You show

openai.TTS.with_azure

— confirm this and the deployment name you configured.) • Can you paste the full traceback text that follows the lines in

livekit/agents/tts/tts.py

? The trace will show whether it’s an auth, timeout, or model error. Immediate, safe steps you can take now 1) Verify Azure TTS auth & endpoint • Ensure

AZURE_OPENAI_API_KEY

and

AZURE_OPENAI_ENDPOINT

(or the variables your code uses) are set in the agent process environment and accessible to the runtime. • Confirm the Azure deployment name and model/voice exist and match the values in

openai.TTS.with_azure(model="tts", voice="nova", api_version="2024-02-15-preview", azure_endpoint=AZURE_OPENAI_ENDPOINT, api_key=AZURE_OPENAI_API_KEY)

. 2) Test connectivity / call Azure TTS directly • From the agent host, run a quick request using curl or the Azure SDK to the same endpoint/credentials to confirm reachability and response (this isolates network/auth from LiveKit code). 3) Capture the exact TTS error • Paste the full traceback (the exception message and cause). Common outcomes: ◦ Auth error -> invalid key or missing scope ◦ 4xx/5xx from Azure -> model or voice name mismatch or service-side error ◦ Timeout/connection error -> network/DNS issues or blocked egress 4) Reduce retry pressure while debugging • Temporarily increase the TTS retry backoff or reduce retry frequency so retries don’t create a retry storm. • If your TTS plugin supports configuring backoff, set longer intervals while you troubleshoot. 5) Mitigate worker load oscillation • Short-term: raise AgentServer

load_threshold

(e.g., 0.9) only if the host CPU/memory can handle it. • Medium/long-term: run multiple agent workers with the same

agent_name

to distribute jobs horizontally; this is the recommended fix for spikes. • Consider offloading TTS to a separate dedicated worker or hosted TTS service if you run local heavy models. 6) Small code/config checks in your snippet • Your

openai.TTS.with_azure(...)

call looks syntactically fine — confirm you pass

azure_endpoint

and

api_key

(or environment equivalents). If you rely solely on env vars, verify the process sees them. • You prewarm VAD; ensure you also prewarm any heavy TTS model instances if the plugin supports it to avoid initialization cost per job. If you send: • The full TTS traceback text • Confirmation of the Azure deployment name and whether you’re using env vars or passing credentials directly • Host CPU/RAM and whether other heavy processes are running I’ll pinpoint the exact fix (auth change, endpoint/model correction, or scaling/config snippet to change). Sources: Agents Framework | livekit/livekit_composite | DeepWiki | Azure OpenAI TTS plugin guide | LiveKit docs | Agents (Python) | livekit/livekit_composite | DeepWiki | Deploy and scale agents on LiveKit Cloud | LiveKit Pricing