https://livekit.io logo
Join Slack
Powered by
Could I setup a stt with an openairealtime model. ...
# ask-ai
d
Could I setup a stt with an openairealtime model. From which one would I get the voice from in this case? Here is an example:
Copy code
session = AgentSession(
                stt = openai.STT(model="gpt-4o-transcribe"),  
                llm = openai.realtime.RealtimeModel(model="gpt-4o-realtime-preview", voice=voice),
                tts = openai.TTS(voice=voice),
                vad = vad
            )
t
Thanks for reaching out, W. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
You can set up speech-to-text (STT) with an OpenAI realtime model in a LiveKit 
AgentSession
. Your configuration is valid, and any combination of STT, LLM, TTS, or realtime API can be used in an 
AgentSession
. Regarding the voice audio source: the realtime model's audio output is controlled by its 
modalities
parameter. By default, the realtime model outputs both text and audio. If you want to use a separate TTS (text-to-speech) for the voice, you should configure the realtime model to use only text modality. This ensures that the audio comes from your specified TTS component, not the realtime model. To use the TTS as the voice source, set up your session like this:
Copy code
session = AgentSession(
    stt=openai.STT(model="gpt-4o-transcribe"),
    llm=openai.realtime.RealtimeModel(
        model="gpt-4o-realtime-preview",
        voice=voice,
        modalities=["text"]  # Use text-only to rely on separate TTS
    ),
    tts=openai.TTS(voice=voice),
    vad=vad
)
This configuration ensures that the voice audio is generated by the TTS component, not the realtime model. Sources: OpenAI STT integration guide | LiveKit Docs | livekit-agents/livekit/agents/voice/agent_session.py | OpenAI Realtime API integration guide | LiveKit Docs
d
Will thi impact the performances of the realtime model ?
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!
d
Ok and what are the benefits of adding an STT model ?
Open in Slack
PreviousNext
Powered by