I get this issue everytime the agent transfers to another one. Is there something I'm doing wrong?

2025-06-01 08:27:07,172 - WARNING livekit.plugins.openai - received text-only response from realtime API 
2025-06-01 08:27:07,257 - WARNING livekit.plugins.openai - trying to recover from text-only response {"retries": 1}
2025-06-01 08:27:08,012 - WARNING livekit.plugins.openai - trying to recover from text-only response {"retries": 2}
2025-06-01 08:27:08,892 - WARNING livekit.plugins.openai - trying to recover from text-only response {"retries": 3}
2025-06-01 08:27:09,649 - WARNING livekit.plugins.openai - trying to recover from text-only response {"retries": 4}
2025-06-01 08:27:10,398 - WARNING livekit.plugins.openai - trying to recover from text-only response {"retries": 5}
2025-06-01 08:27:11,242 - ERROR livekit.plugins.openai - failed to recover from text-only response {"retried_times": 5}
2025-06-01 08:27:11,892 - ERROR livekit.agents - Error in _realtime_reply_task

Hello, Team I just start my voice agent using OpenAI realtime S2S model, My Simple code which is from voice agent quick start guide is showing me this error , .plugins.openai.realtime.realtime_model.RealtimeModel' error=APIConnectionError('OpenAI S2S connection closed unexpectedly') recoverable=False 2025-06-05 105209,018 - ERROR livekit.agents - AgentSession is closing due to unrecoverable error livekit.agents._exceptions.APIConnectionError: OpenAI S2S connection closed unexpectedly 2025-06-05 105209,018 - ERROR livekit.plugins.openai - Error in _recv_task Traceback (most recent call last): File "P:\ALLIED BANK Professional DATA\ABL MUAWIN CUSTOMER_PRODUCTION CODE\TEST_VOICE_AGENT\agent_voice\venv\Lib\site-packages\livekit\agents\utils\log.py", line 16, in async_fn_logs return await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "P:\ALLIED BANK Professional DATA\ABL MUAWIN CUSTOMER_PRODUCTION CODE\TEST_VOICE_AGENT\agent_voice\venv\Lib\site-packages\livekit\plugins\openai\realtime\realtime_model.py", line 581, in _recv_task raise error Exception: OpenAI S2S connection closed unexpectedly What is the Issue, why facing this error opently

OpenAI has just released an update for their realtime 4o mode: https://platform.openai.com/docs/models/gpt-4o-realtime-preview "We just released an updated snapshot of our speech-to-speech model, now available as gpt-4o-realtime-preview-2025-06-03 in the Realtime API and gpt-4o-audio-preview-2025-06-03 in the Chat Completions API. This update addresses top pieces of user feedback: the model follows instructions more reliably, handles interruptions better, and makes tool calls more consistently. We’d love to hear what you think, especially if these areas have been frustrating in the past. What else would you like to see improved in our speech-to-speech models? Please share your thoughts in this Dev Community thread."

Hi anyone experiencing this issue, same code base was working this morning

- INFO livekit.agents - registered worker {"id": "AW_hdt4bkgLCnTR", "url": "<wss://XXXXXX-XXXX.livekit.cloud>", "region": "France", "protocol": 16}
- INFO livekit.agents - received job request {"job_id": "AJ_jDwdep5K69Eh", "dispatch_id": "", "room_name": "thread-_+XXXXXXXXXFHvijNdHGBY", "agent_name": "", "resuming": false}
- INFO livekit.agents - initializing process {"pid": 68620}
- INFO livekit.agents - process initialized {"pid": 68620, "elapsed_time": 0.9}
- DEBUG asyncio - Using selector: KqueueSelector {"pid": 68620, "job_id": "AJ_jDwdep5K69Eh"}
- DEBUG livekit.agents - http_session(): creating a new httpclient ctx {"pid": 68620, "job_id": "AJ_jDwdep5K69Eh"}
- DEBUG livekit.agents - start reading stream {"participant": "sip_+33662230502", "source": "SOURCE_MICROPHONE", "pid": 68620, "job_id": "AJ_jDwdep5K69Eh"}
- ERROR livekit.agents - Error in _realtime_reply_task
....

livekit.agents.llm.realtime.RealtimeError: generate_reply timed out. {

If I'm using Gemini Real-Time or any other real-time API when I'm running agent.py locally, is there a way to interact with the agent with text and not just voice?

do you guys support the new gemini live with function calls? i didn't see function calls as an option in the docs.

Hey guys,

interrupt_response=False

isn't working for me to disable the ability to interrupt the agent. I'm using OpenAI's realtime model, with ElevenLabs TTS. Am I doing anything wrong in the configuration?

return openai.realtime.RealtimeModel(
            model=model,
            temperature=temperature,
            speed=speed,
            modalities=["text"],
            input_audio_noise_reduction=InputAudioNoiseReduction(
                type="near_field"
            ),
            input_audio_transcription=InputAudioTranscription(
                model="gpt-4o-mini-transcribe",
            ),
            turn_detection=TurnDetection(
                type="semantic_vad",
                eagerness="auto",
                interrupt_response=False,
            ),
        )

---

session = AgentSession[OutboundCallData](
            stt=NOT_GIVEN,
            llm=llm_engine,
            tts=tts_engine,
            vad=vad,
            userdata=data,
            mcp_servers=mcp_servers,
            user_away_timeout=5,
        )

"In VoiceAgent, I have a Firebase real-time variable tempAction. I want to call self.session.update_agent(VisionAssistant()) as soon as tempAction becomes 1. How can I do this?"

Hello Everyone I'm building the ai voice agent for my business, I was using openAi realtime model witn no problem, untill mid of july the agent start to be stuped and it doesn't give me the same results as before, some times it talk with un undertsndable language and it can't detect correctly the data gives to it from project meta data and injected into chat as system role, while it was perfect before , aslo the function tools no more working , and when I set it , the agent looks loost and doesn't answer the user input and keep answering randomally here is how I configuring it # 4. Fetch the metadata and update the agent's context BEFORE starting the session try: # Solution 4: Add timeout to metadata fetching fetcher = ProjectMetadata(access_token) metadata_messages = await asyncio.wait_for( fetcher.fetch_metadata(project_id), timeout=15.0 # 15 second timeout ) if metadata_messages: chat_ctx = assistant_agent.chat_ctx.copy() for message in metadata_messages: chat_ctx.add_message(role="system", content=message) await assistant_agent.update_chat_ctx(chat_ctx) except asyncio.TimeoutError: print("Warning: Metadata fetching timed out, proceeding without metadata") except Exception as e: print(f"Failed to fetch or apply project metadata: {e}") # 5. Create session with timeout and better error handling try: session = AgentSession( llm=openai.realtime.RealtimeModel( modalities=['text', 'audio'], temperature=0.6, model='gpt-4o-realtime-preview', max_session_duration=5000, turn_detection=TurnDetection( type="server_vad", threshold=0.5, prefix_padding_ms=300, silence_duration_ms=500, create_response=True, interrupt_response=True, ) ), tts=openai.TTS(model="tts-1-hd", voice="sage"), vad=ctx.proc.userdata["vad"], min_endpointing_delay=0.2, max_endpointing_delay=3.0, ) Anyone has the same issue ? any Help on this topic?

Hey team there's a major issue with the openai realtime model responding in random languages when there's not a lot of context and, it happens in around 25% of the conversations! cc @refined-appointment-81829

The system prompt isn’t applied until after the first user message, so the greeting message doesn’t follow the language instruction. Is there a way to set a default language for the greeting itself? Currently, it greets user in random languages

Hey I'm using google realtime api for building speech to speech model I want to achieve that first the Bot say "Thank you for your time. Have a great day!" & than end the call: --- it's not working when calling inside of the func session.generate_reply(insturctions="")

await for session.generate_reply(instruction= "Tell the User:Thank you for your time. Have a great day! ")

await ctx.session.aclose()
async with api.LiveKitAPI() as lkapi:
        await lkapi.room.delete_room(DeleteRoomRequest(room_name=ctx.userdata['room_name']))

also I have tried this :

this one is working 
return "Thank you for your time. Have a great day!"

the Problem is it's Just saying it but never end the call SO I tried to use it retruning the str & calling another func

retrun "Thank you for your time. Have a great day!",await  end_room(ctx)

async def end_room(ctx):
    await ctx.session.aclose()
async with api.LiveKitAPI() as lkapi:
        await lkapi.room.delete_room(DeleteRoomRequest(room_name=ctx.userdata['room_name']))

Has anyone tried new OpenAI realtime api? Is it significantly better?

Noticed that support for image inputs is not yet implemented for the OpenAI Realtime models. Im happy to work on this feature and opened an issue for it here. @refined-appointment-81829

hey @refined-appointment-81829 i think

conversation_item_added

event isn't working with realtime models. could you confirm if that works for you? both the user inputs and the agent inputs. in my case only the agent inputs work and

user_input_transcribed

doesn't have anything in

transcript

for some reason

which noise cancelation is better OpenAI's or Krisp via livekit? cc @refined-appointment-81829

any ideas to get detailed usage https://livekit-users.slack.com/archives/C07FY8WHGPM/p1756799670336879

How do I connect to the EU region? I get this

aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url='<wss://eu.api.openai.com/realtime?model=gpt-4o-realtime-preview-2025-06-03>'

I’m trying to use the

base_url="<https://eu.api.openai.com>"

option

We are noticing that after every Agent utterance, we are thrown an error, has anyone else experienced this (Agent NodeJS V1, and GPT5 Realtime Model):

ERROR (48861): Invalid state: ReadableStream is locked NR-LINKING

Does anybody have experience configuring their agent to only allow interruptions with certain pre-defined phrases? As voice detection can be inconsistently sensitive I am exploring options for only allowing interruptions with clearly deliberate phrases.

Hey team! Does LikeKit already support the new

gtp-realtime

model? I can’t reproduce the emotions capabilities available in the new model.

Also do I need at all Deepgram to transcribe STT if I'm using openai realtime model? How to save the chat history for example.

How to reduce latency when I am using Realtime model as Model in AgentSession with VAD added and integrated Livekit tools to call LangGraph Function Nodes. Agent takes noticeable time to respond,

When i use gpt-realtime model i am getting error

type='realtime_model_error' timestamp=1757434133.171836 label='livekit.plugins.openai.realtime.realtime_model.RealtimeModel' error=APIError('OpenAI Realtime API response failed with error type: server_error') recoverable=True

But i can see the session is created in OpenAI tracing dashboard

How to add Fallback if OpenAI realtime throws internal server error.

type='realtime_model_error' timestamp=1757434133.171836 label='livekit.plugins.openai.realtime.realtime_model.RealtimeModel' error=APIError('OpenAI Realtime API response failed with error type: server_error') recoverable=True

<livekit.plugins.openai.realtime.realtime_model.RealtimeModel object at 0x7f24ad24a710>: type='realtime_model_error' timestamp=1757511738.2698364 label='livekit.plugins.openai.realtime.realtime_model.RealtimeModel' error=AssertionError() recoverable=False

set the channel topic: Conversations for this channel will be moving to Agents

While using Openai Realtime model along with AgentSession and livekit tools traceable functions with LangGraph it has noticeable latency.. How to fix this one?

set the channel topic: Aske Realtime questions in #C07FY8WHGPM

Aske Realtime questions in #C07FY8WHGPM

archived the channel