https://livekit.io logo
Join Slack
Powered by
is there a way to use the OpenAI RealtimeModel for...
# ask-ai
m
is there a way to use the OpenAI RealtimeModel for STT + LLM, while having a separate model handle the TTS part? Based on my testing, if I set modalities to 
["text"+"audio"]
it overrides my TTS model with the OpenAI model.
t
Thanks for reaching out, Anas. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
You can use OpenAI's RealtimeModel for STT and LLM while using a separate model for TTS in LiveKit. To achieve this, configure the RealtimeModel with a text-only response modality and include a separate TTS plugin in your 
AgentSession
configuration. This ensures the RealtimeModel handles speech comprehension, while your chosen TTS provider manages speech output. Here’s an example configuration:
Copy code
session = AgentSession(
    llm=openai.realtime.RealtimeModel(modalities=["text"]), 
    tts=cartesia.TTS()  # Or any other TTS plugin
)
Make sure to set 
modalities=["text"]
for the RealtimeModel to avoid overriding your TTS model. Let me know if you need further assistance!
m
so if I don't specify an 
stt
argument in the 
AgentSession
, it somehow defaults to the RealtimeModel's STT capabilities?
t
Hey there! When you're using OpenAI's RealtimeModel, if you don’t specify an 
stt
argument in the 
AgentSession
, it’ll automatically use the RealtimeModel’s built-in STT features. That’s because the RealtimeModel nicely handles both STT and LLM functions all by itself. But, if you’re working with LiveKit’s turn detector, leaving out the 
stt
argument will lead to an error. That’s because the turn detector needs a separate STT plugin to work properly. So, just make sure to include the 
stt
argument with a valid STT plugin if you’re relying on LiveKit’s turn detection instead of the RealtimeModel’s built-in capabilities. Hope that helps!
Open in Slack
PreviousNext
Powered by