model = openai realtime RealtimeModel voice = shimmer temper LiveKit Community #ask-ai

model = openai.realtime.RealtimeModel( voice="...

strong-pizza-31661

07/21/2025, 5:55 PM

model = openai.realtime.RealtimeModel( voice="shimmer", temperature=0.8, #model="gpt-4o-realtime-preview-2025-06-03", # Tell Whisper to expect Turkish (“tr”) audio _input_audio_transcription_=None, _turn_detection_=None, modalities=["text"] ) # Create AgentSession with the multimodal model session = AgentSession( llm=model, tts=elevenlabs.TTS( language=LANG_MODE, _voice_id_="cgSgspJ2msm6clMCkdW9", _streaming_latency_=3, ), vad=silero.VAD.load(), stt=deepgram.STT(model="nova-2", language=LANG_MODE), _#turn_detection=MultilingualModel(),_ ) is this combination of the multimodal and non multimodal work? is this allowed?

tall-airline-16756

07/21/2025, 5:55 PM

Thanks for reaching out, Rhythm. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

tall-airline-16756

07/21/2025, 5:56 PM

You can use a non-multimodal model (with

modalities=["text"]

) together with a multimodal

AgentSession

configuration that includes TTS, VAD, and STT components. This combination is explicitly supported and documented. The

AgentSession

constructor is designed to accept separate components for different modalities. You can mix and match these as needed, for example:

Copy code

session = AgentSession(
    llm=openai.realtime.RealtimeModel(modalities=["text"]),
    tts=elevenlabs.TTS(...),
    vad=silero.VAD.load(),
    stt=deepgram.STT(...)
)

This approach is useful if you want to use a different TTS provider, or if you want to maintain control over speech output while still benefiting from realtime speech comprehension. The same pattern is supported across different realtime model providers. This configuration is allowed and will work as intended. Sources: Realtime model integrations | LiveKit Docs | OpenAI Realtime API integration guide | LiveKit Docs | livekit-agents/livekit/agents/voice/agent_session.py | Azure OpenAI Realtime API and LiveKit | LiveKit Docs | Gemini Live API integration guide | LiveKit Docs

5 Views

Open in Slack

Previous Next