and we need to have one thing in place, whenver it hits, search_knowledge_base tool, it takes up to 3s to pull the data. and in that timeframe, the agent is dead silence. user doesnt know that it is checking. check in livekit sdk, that can we add a filler session.say just before it hits the tool. it is possible, check and let me kow

Thanks for reaching out, Samuel. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

works for realtime models?

Yes — the filler

session.say

approach works with real‑time models. Key considerations: • Use non‑blocking

session.say()

so the filler plays while the realtime model or tool runs. • For streaming TTS, ensure your SDK supports low‑latency TTS streaming; otherwise pre‑synthesize the clip for instant playback. • Use the returned

SpeechHandle

to stop the filler if the realtime response arrives quickly to avoid overlap. • Keep the filler short and neutral to preserve UX. Tell me which LiveKit SDK/language you’re using (JS, Python, or other) and whether you prefer streaming TTS or pre‑synthesized audio, and I’ll provide a concise code example. , with interactive elements

what of session.generatereply. will that work?

Responding to your question about using

session.generateReply

. Short answer: No —

session.generateReply

is intended to produce the agent's reply after processing (including after a tool finishes). It isn't designed as a short filler while a tool is running. Recommended approach: • Use

session.say()

(or

session.speak()

if available) as a non-blocking call immediately before invoking

search_knowledge_base

to play a quick filler. • Start the search, then when the tool returns call

session.generateReply()

to build and play the actual reply. • To avoid overlap: capture the

SpeechHandle

from

session.say()

and stop it if the tool responds quickly. • For lowest latency, pre-synthesize the filler audio and pass it to `session.say(audio=...)`; otherwise use streaming TTS if supported. If you want, tell me your SDK/language (JS, Python, etc.) and I’ll provide a concise code snippet showing

session.say()

+ non-blocking search +

session.generateReply()

. , with interactive elements

i use python.

Thanks — responding to your request for a concise Python example showing a filler

session.say()

before

search_knowledge_base

, handling the

SpeechHandle

, and then using

session.generate_reply()

. Notes up front: • Use non-blocking

session.say()

so playback starts while the search runs. • Capture the returned

SpeechHandle

to stop the filler if the tool returns quickly. • For lowest latency, pass pre-synthesized audio to `session.say(audio=...)`; otherwise TTS is fine. • This pattern works with realtime models. Example (concise, illustrative):

# assume `session` is your LiveKit session object and `search_knowledge_base` is a callable

# 1) play filler non-blocking
speech_handle = session.say("Let me check that for you.")  # do not await

# 2) start the long-running tool (run concurrently)
result = search_knowledge_base(query)  # blocking call; run in background or thread if needed

# If the tool can be awaited/async, you can run both concurrently:
# import asyncio
# filler = session.say("Let me check that for you.")
# result = await asyncio.create_task(search_knowledge_base(query))

# 3) stop filler if still playing (optional)
if speech_handle and not speech_handle.finished:
    speech_handle.stop()  # or speech_handle.cancel()

# 4) generate and play the final reply using session.generate_reply()
reply = session.generate_reply(result)  # builds reply from tool output
session.say(reply)  # play final reply (await if you want to ensure completion)

If you want, I can: • Convert this to an async/await pattern for asyncio, • Show how to use pre-synthesized audio (

audio=

param), or • Give a thread-based example if your

search_knowledge_base

is synchronous. Which would you prefer?, with interactive elements

I'm using livekit-agents 1.5.6 with Python. My AgentSession is configured like this: ▎ ▎ session = AgentSession( ▎ llm=openai.realtime.RealtimeModel( ▎ model="gpt-realtime", ▎ voice="marin", ▎ turn_detection=TurnDetection(type="semantic_vad", eagerness="medium"), ▎ ), ▎ # NO tts= parameter ▎ # NO stt= parameter ▎ ) ▎ ▎ I want to play a short filler phrase ("Let me check that for you.") at the start of a @function_tool that does a slow REST API call (~3s), so the user doesn't sit in silence. Hard constraints: ▎ ▎ 1. I cannot add a TTS plugin to the AgentSession. ▎ 2. I cannot use pre-recorded WAV files (no static audio assets in the repo). ▎ 3. I do not want to pre-synthesise audio at worker startup either. ▎ 4. I want the filler in the same voice as the realtime model ("marin"). ▎ ▎ Three specific questions: ▎ ▎ Q1. Will context.session.say("Let me check that for you.") work in this exact stack (no TTS plugin, no audio= param)? If not, what is the exact error or behavior? ▎ ▎ Q2. Is there any LiveKit API that lets the realtime model render a specific predetermined short phrase using its own internal TTS, without going through the full LLM inference cycle (i.e., bypassing the ▎ response model so I get sub-200ms playback start)? ▎ ▎ Q3. If Q1 and Q2 are both no, please confirm: the only options for spoken filler audio in this stack are (a) add a TTS plugin to AgentSession, (b) pre-synthesise audio at startup, or (c) use pre-recorded files. ▎ Are there any other documented options I'm missing? ▎ ▎ Please cite the relevant docs page URLs for each answer.

Thanks — I’ll answer your three questions directly for your exact stack (livekit-agents 1.5.6, Python AgentSession with a realtime LLM, no

tts=

plugin). Q1 — Will

context.session.say("Let me check that for you.")

work as-is? • No.

session.say()

requires a configured TTS provider. With no

tts=

plugin and no

audio=

param,

session.say(text)

cannot synthesize audio and will fail (no local TTS engine to produce speech). Q2 — Is there an API to make the realtime model render a predetermined phrase instantly (bypass full LLM inference) for sub-200ms playback? • No. There is no documented LiveKit API that lets you directly ask the realtime model to synthesize a supplied text string immediately without entering the full response/inference cycle. Generating audio from the realtime model requires the normal response pipeline (e.g.,

generate_reply()

), which adds noticeable latency and is not a guaranteed verbatim TTS bypass. Q3 — If Q1 and Q2 are no, confirm the available options under your constraints. • Correct: under your hard constraints, the documented options are: ◦ (a) Add a TTS plugin to the AgentSession so

session.say()

can synthesize text. ◦ (b) Pre-synthesize (cache) audio at startup and play it via

session.say(audio=...)

. ◦ (c) Use pre-recorded audio assets (WAVs) and play them. • There are no other documented ways to get low-latency spoken filler in the same

marin

voice without one of the above. Relevant docs: • Agent speech / audio: https://docs.livekit.io/agents/multimodality/audio/ • Realtime models overview: https://docs.livekit.io/agents/models/realtime/ • Audio customization / pre-synth: https://docs.livekit.io/agents/multimodality/audio/customization/ If you can relax any constraint, tell me which one and I’ll give the minimal code path (e.g., how to add the OpenAI TTS plugin to AgentSession, or an async pattern using

generate_reply()

if you’ll accept the added latency). Sources: Agent speech and audio | LiveKit Documentation | Realtime models overview | LiveKit Documentation | Audio customization | LiveKit Documentation, with interactive elements