https://livekit.io logo
Join Slack
Powered by
does `openai.realtime.RealtimeModel` trigger an ev...
# ask-ai
m
does 
openai.realtime.RealtimeModel
trigger an event to 
on_user_turn_completed
?
t
Thanks for reaching out, Anas. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Hey there! Just a quick heads-up: the 
openai.realtime.RealtimeModel
doesn’t directly trigger the 
on_user_turn_completed
event. When server-side turn detection is turned on (which is the default for RealtimeModel), the 
on_user_turn_completed
callback gets skipped entirely. If you’d like to use 
on_user_turn_completed
with a RealtimeModel, here’s what you need to do: 1. Turn off server-side turn detection in the RealtimeModel by setting 
turn_detection=None
. 2. Set up turn detection in the agent itself, using VAD or another method. Here’s a quick example:
Copy code
# Disable turn detection in RealtimeModel
llm = openai.realtime.RealtimeModel(
    voice="alloy",
    turn_detection=None  # Disable server-side turn detection
)

# Configure turn detection in the agent session
session = AgentSession(
    turn_detection="vad"  # Use VAD for turn detection
)
Doing it this way makes sure your 
on_user_turn_completed
callback runs just like you expect. Hope this helps! Sources: livekit-agents/livekit/agents/voice/agent_activity.py | Pipeline nodes and hooks | LiveKit Docs
m
if that's the case, is it better to use 
on_user_turn_completed
or 
tts_node
?
t
When you're trying to decide between 
on_user_turn_completed
and 
tts_node
with 
openai.realtime.RealtimeModel
, it really comes down to what you need for your specific situation: • Go with 
on_user_turn_completed
 if you want to work with or tweak the user’s input right after their turn. This is great for adding extra context, doing things like RAG lookups, or getting the agent’s response ready. • Choose 
tts_node
 if you need to customize text-to-speech behavior. That said, with realtime models, this is usually less important since they handle speech synthesis on their own. In most cases when you’re working with realtime models, 
on_user_turn_completed
tends to be the smoother choice because it fits right into the flow of a real-time conversation. Sources: Voice pipeline nodes | LiveKit Docs | Pipeline nodes and hooks | LiveKit Docs | livekit-plugins/livekit-plugins-turn-detector/README.md
m
what turn detection model should I use?
Copy code
TurnDetectionMode = Union[Literal["stt", "vad", "realtime_llm", "manual"], _TurnDetector]
"""
The mode of turn detection to use.

- "stt": use speech-to-text result to detect the end of the user's turn
- "vad": use VAD to detect the start and end of the user's turn
- "realtime_llm": use server-side turn detection provided by the realtime LLM
- "manual": manually manage the turn detection
- _TurnDetector: use the default mode with the provided turn detector

(default) If not provided, automatically choose the best mode based on
    available models (realtime_llm -> vad -> stt -> manual)
If the needed model (VAD, STT, or RealtimeModel) is not provided, fallback to the default mode.
"""
t
Hey there! The best 
TurnDetectionMode
to use with 
openai.realtime.RealtimeModel
is 
"realtime_llm"
. This mode taps into the server-side turn detection from the realtime LLM, which is really great at accurately figuring out when a turn is complete. Here’s a quick rundown for you: • Recommended Mode: 
"realtime_llm"
is made just for realtime models and nails turn detection with high accuracy. • Fallback: If 
realtime_llm
isn’t available, no worries—the system will automatically switch to other modes like 
vad
or 
stt
. • Configuration: You can also tweak settings like 
threshold
, 
silence_duration_ms
, and 
eagerness
to get the behavior just right for your needs. Here’s an example to make it clearer:
Copy code
session = AgentSession(
    llm=openai.realtime.RealtimeModel(
        turn_detection=TurnDetection(
            type="server_vad",
            threshold=0.5,
            prefix_padding_ms=300,
            silence_duration_ms=500,
            create_response=True,
            interrupt_response=True,
        )
    ),
    turn_detection="realtime_llm"
)
This setup is a solid way to get the best performance out of the RealtimeModel. Hope this helps! Sources: Agent Testing & Evaluation | livekit/livekit_composite | DeepWiki | Agents Core Architecture | livekit/livekit_composite | DeepWiki | OpenAI Realtime API integration guide | LiveKit Docs
m
but if I use that, how can I intercept the LLM instructions (before generating) and update those instructions before they are passed to the LLM?
t
If you want to catch and tweak the LLM instructions before they get to the LLM, here are a couple of friendly ways to do it: 1. Override the `llm_node`: Think of the 
llm_node
as your chance to peek into the chat context and make any changes before it heads off to the LLM. By overriding this node, you can shape how the LLM responds to fit your needs. 2. Use the 
update_instructions
method: This handy method lets you refresh the agent's instructions and, if you’re working with a realtime session, it updates those instructions there too. Here’s a quick example:
Copy code
await agent.update_instructions(new_instructions)
With these tools, you’ve got the flexibility to adjust instructions on the fly, making sure the LLM gets exactly what it needs to work its magic. Sources: livekit-agents/livekit/agents/voice/agent_activity.py
m
I have the following config, but it's still not triggering the llm_node:
Copy code
session: AgentSession = AgentSession(
    turn_detection="realtime_llm",
)
Copy code
agent._llm = openai.realtime.RealtimeModel(
    modalities=["text"],
    turn_detection=None
)

agent._vad = None
t
Hey there! To get to the bottom of why the 
llm_node
isn’t triggering, could you please check a few things for me? 1. Is your 
RealtimeModel
set up with 
create_response=True
and 
interrupt_response=True
in the turn detection settings? 2. Does your 
AgentSession
have an STT plugin included? 3. And is the 
allow_interruptions
setting turned on with 
True
? Once we have these details, it’ll be much easier to figure out what’s going on. Thanks so much for checking! Sources: livekit-agents/livekit/agents/voice/agent_activity.py | LiveKit turn detector plugin | LiveKit Docs | LiveKit turn detector plugin | LiveKit Docs
2 Views
Open in Slack
PreviousNext
Powered by