https://livekit.io logo
Join Slack
Powered by
how session say works with realtime llm?
# ask-ai
w
how session say works with realtime llm?
t
Thanks for reaching out, Test. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about how 
session.say()
works with realtime LLMs. Short answer • 
session.say()
requires a TTS plugin to synthesize audio. If you only have a realtime LLM (one that directly handles audio IO), 
session.say()
will not produce audio by itself. • If you want agent-initiated spoken output using a realtime LLM alone, use 
generate_reply()
(the realtime model’s built-in audio output) or add a separate TTS plugin to the session so 
session.say()
can use that TTS. What’s happening and your options • Realtime LLMs handle audio input/output natively, so they can both listen and speak without a separate TTS. Because 
session.say()
is designed to call a configured TTS plugin, it won’t work if no TTS is present. • Two practical approaches: a. Include a TTS plugin in the AgentSession and keep using 
session.say()
for synthesized speech. • This is a half-cascade (half-duplex) setup: realtime model may handle audio input but outputs text; the TTS plugin produces the final audio. • Example pattern: configure 
llm
for text (or modalities that exclude native audio output) and set 
tts
to a TTS model, then call 
await session.say("...")
. a. Use 
generate_reply()
with the realtime LLM when you want the model itself to speak. • 
generate_reply()
sends instructions or user input to the realtime model and the model’s native audio generation produces speech without a separate TTS. If you’d like, tell me: • Which realtime LLM you’re using (OpenAI Realtime, Nova Sonic, Gemini Live, etc.), and • Whether you prefer to keep a separate TTS or rely entirely on the realtime model’s native audio. I can then show a short example for your specific setup and the exact code changes you’ll need. Sources: Agent speech and audio | LiveKit Documentation | Realtime models overview | LiveKit Documentation | Amazon Nova Sonic integration guide | LiveKit Documentation
w
so tts node will not be utilized by session if llm give audio output ?
t
Short answer: Correct — if a realtime LLM is producing audio output natively, the session will not use a separate TTS plugin for that audio. The TTS node is only used when 
session.say()
(or another flow that targets the configured TTS) is invoked. More detail and what that means for you • Realtime LLMs can handle audio input and audio output themselves. When you call methods that send instructions to the realtime model (for example 
generate_reply()
), the model’s native audio generation will produce speech without calling your configured TTS. • 
session.say()
is specifically designed to route text to a configured TTS plugin. If no TTS is configured, 
session.say()
won’t produce audio even if a realtime LLM is present. • If you configure both a realtime LLM and a TTS in the same session, you can choose which to use: ◦ Use 
generate_reply()
to let the realtime model speak using its built-in audio. ◦ Use 
session.say()
to force the session to synthesize speech via the TTS plugin. Which approach to pick • Use the realtime model’s native audio when you want fewer components and lower latency (model handles end-to-end audio). • Use a separate TTS when you need specific voice control, fine-grained TTS settings, or want consistent TTS across different LLM backends. If you want, tell me: • Which realtime model you're using, and • Whether you want examples for using 
generate_reply()
(realtime audio) or for configuring 
session.say()
with a TTS plugin. I can provide a short code example for your exact setup.
Open in Slack
PreviousNext
Powered by