I am getting the following error: {"message": "starting voice assistant for participant JamesCarter", "level": "INFO", "name": "voice-assistant", "pid": 18, "job_id": "AJ_hiBKuVxGvUDs", "room_id": "RM_p589UgkNUXGa", " {"message": "RoomInputOptions and RoomOutputOptions are deprecated, use RoomOptions instead", "level": "WARNING", "name": "livekit.agents", "pid": 18, "job_id": "AJ_hiBKuVxGvUDs", "ro {"message": "failed to synthesize speech, retrying in 0.1s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "process initialized", "level": "INFO", "name": "livekit.agents", "pid": 32, "elapsed_time": 3.36, "timestamp": "2025-12-29T150529.396362+00:00"} {"message": "worker is at full capacity, marking as unavailable", "level": "INFO", "name": "livekit.agents", "load": 0.9957419999999999, "threshold": 0.7, "timestamp": "2025-12-29T15: {"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/tts/tts.py\", {"message": "STT metrics", "level": "INFO", "name": "livekit.agents", "model_name": "nova-3", "model_provider": "Deepgram", "audio_duration": 4.45, "pid": 18, "job_id": "AJ_hiBKuVxGvU {"message": "worker is below capacity, marking as available", "level": "INFO", "name": "livekit.agents", "load": 0.42745039999999984, "threshold": 0.7, "timestamp": "2025-12-29T1505 {"message": "Error in _tts_inference_task\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/utils/log.py\", line 16, in as {"message": "STT metrics", "level": "INFO", "name": "livekit.agents", "model_name": "nova-3", "model_provider": "Deepgram", "audio_duration": 5.0, "pid": 18, "job_id": "AJ_hiBKuVxGvUD {"message": "EOU metrics", "level": "INFO", "name": "livekit.agents", "end_of_utterance_delay": 0.55, "transcription_delay": 0.47, "pid": 18, "job_id": "AJ_hiBKuVxGvUDs", "room_id": " {"message": "failed to generate LLM completion, retrying in 0.1s\nTraceback (most recent call last):\n File \"/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/llm/llm How do I fix it?

Thanks for reaching out, shreyasraoinna. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

For TTS I am using openai. for LLM I am using azure Server.rtc_session() async def entrypoint(ctx: JobContext): await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY) participant = await ctx.wait_for_participant() logger.info(f"starting voice assistant for participant {participant.identity}") if participant.identity: username = participant.identity else: # Adding this line so that conversations will work in sandbox frontend username = "PaulSmith" system_instructions, vector_store_ids = get_system_instructions(username) voice_instructions = "You are a voice agent. Ensure that your responses are short and concise.\n" system_instructions = voice_instructions + system_instructions vad_instance = ctx.proc.userdata.get("vad") if vad_instance is None: # fallback (should not happen if prewarm ran) vad_instance = silero.VAD.load() agent = Agent( instructions=system_instructions, # vad=vad_instance, stt=deepgram.STT(), llm=openai.LLM.with_azure(azure_deployment="gpt-4o", model="gpt-4o", azure_endpoint=AZURE_OPENAI_ENDPOINT, api_key=AZURE_OPENAI_API_KEY, api_version="2024-12-01-preview"), # tts=openai.TTS(model="tts-1", voice="sage"), tts=openai.TTS.with_azure(model="tts", voice="nova", api_version="2024-02-15-preview", instructions="Speak slowly in a calm and soothing voice."), # llm=openai.LLM(model="gpt-4o"), # tts=openai.TTS(model="tts-1",voice="sage"), tools=[#query_info, email_alert_for_suicide,email_alert_to_property_manager,email_alert_to_entitlements_specialist,email_alert_to_employment_specialist], ) session = AgentSession(userdata=MySessionInfo(user_name=username, vector_store_id=vector_store_ids[0]),vad=vad_instance,) # log metrics as they are emitted, and total usage after session is over usage_collector = metrics.UsageCollector() @session.on("metrics_collected") def _on_metrics_collected(ev: MetricsCollectedEvent): metrics.log_metrics(ev.metrics) usage_collector.collect(ev.metrics) async def log_usage(): summary = usage_collector.get_summary() # chat_history = str(session.history.to_dict()) # write_chat_to_db(chat_history, username=username) logger.info(f"Usage: {summary}") # shutdown callbacks are triggered when the session is over ctx.add_shutdown_callback(log_usage) await session.start(agent=agent, room=ctx.room, room_input_options=RoomInputOptions( # LiveKit Cloud enhanced noise cancellation # - If self-hosting, omit this parameter # - For telephony applications, use

BVCTelephony

for best results noise_cancellation=noise_cancellation.BVC(), ), ) await session.say("Hey, how can I help you today?", allow_interruptions=True)

Thanks — this is very helpful. I’m responding to the parts you provided: your AgentSession code (you’re using OpenAI TTS with Azure and Azure for LLM), the deprecated RoomInputOptions usage, TTS failures, LLM failures, and worker-load warnings. Below are focused fixes and the minimal code changes to try next. Key fixes to apply 1. Replace deprecated options • Use

room_options=RoomOptions(...)

instead of

room_input_options=...

when starting the session. 1. Correct Azure/OpenAI provider usage • Your LLM config looks like an Azure OpenAI client but is passed as

openai.LLM.with_azure(...)

— keep that (it’s fine). For TTS with Azure, use the Azure-specific constructor consistently:

openai.TTS.with_azure(...)

is correct, but ensure the

model

and

api_version

match what your Azure deployment expects (you had

model="tts"

and

api_version="2024-02-15-preview"

). 1. Ensure required environment variables / keys are set • Confirm these are set and correct:

AZURE_OPENAI_ENDPOINT

,

AZURE_OPENAI_API_KEY

. If using OpenAI-native TTS, you’d set

OPENAI_API_KEY

instead. Missing/invalid keys cause the synth/LLM failures. 1. Add connection options (retries/timeouts) to TTS and LLM to reduce transient failures • Use

conn_options

with reasonable

max_retries

and

timeout

for both

llm

and

tts

. 1. Check host load and resource limits • The logs show worker load ~0.995. Reduce concurrent jobs or increase instance resources. High CPU/RAM will cause timeouts in TTS/LLM calls. Minimal corrected code (apply these changes)

from livekit.agents.voice.room_io import RoomOptions
from livekit.agents import ConnectionOptions

agent = Agent(
    instructions=system_instructions,
    stt=deepgram.STT(),
    llm=openai.LLM.with_azure(
        azure_deployment="gpt-4o",
        model="gpt-4o",
        azure_endpoint=AZURE_OPENAI_ENDPOINT,
        api_key=AZURE_OPENAI_API_KEY,
        api_version="2024-12-01-preview",
        conn_options=ConnectionOptions(max_retries=3, timeout=30.0),
    ),
    tts=openai.TTS.with_azure(
        model="tts",               # confirm this matches your Azure deployment name
        voice="nova",
        api_version="2024-02-15-preview",
        api_key=AZURE_OPENAI_API_KEY,
        azure_endpoint=AZURE_OPENAI_ENDPOINT,
        conn_options=ConnectionOptions(max_retries=3, timeout=30.0),
        instructions="Speak slowly in a calm and soothing voice.",
    ),
    tools=[...],
)

session = AgentSession(
    userdata=MySessionInfo(...),
    vad=vad_instance,
)

await session.start(
    agent=agent,
    room=ctx.room,
    room_options=RoomOptions(
        noise_cancellation=noise_cancellation.BVC(),
    ),
)

Troubleshooting steps to run now • Test Azure TTS and Azure LLM from a small standalone script on the same host to confirm credentials and network reachability. • Monitor CPU/memory while running a test call; if CPU is near 100% increase instance size or reduce concurrent agent jobs. • If you still see retry messages, capture one full failure stack trace for TTS and one for LLM (the full exception text) and paste it here. If you want, paste the exact values you set for `model`/`deployment` in Azure (redact keys) and confirm whether the agent runs on your own server or LiveKit Cloud — I’ll validate the model names/api_version and suggest any remaining adjustments.

It is working fine on my local machine