does the LiveKit agents SDK support speaking while a tool is executing?

Thanks for reaching out, Kevin. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I'm going to show you what my LLM said so you can tell me if its wrong: What we should be doing differently: The initial acknowledgment should use

session.say()

with a short pre-set phrase, not

generateReply

. This is instant, doesn't create an LLM turn, and can be interrupted. For the progressive filler, same thing --

session.say("Still working on that", { allowInterruptions: true })

. When the tool completes and the LLM generates the real response, it naturally interrupts the filler. 1. The

ctx

parameter in the execute function likely has

wait_for_playout()

-- we should check if the LiveKit JS SDK exposes this, and if so, use it to coordinate. The core problem we saw in the logs (agent saying "Please add it to my cart") is exactly because

generateReply

generates a full LLM response that has no idea it's supposed to be a brief filler -- it generates whatever the model thinks is appropriate, which sometimes is the user's perspective narrated back.

Thanks — I’m responding to your points about using

session.say()

vs

generateReply()

and using

wait_for_playout()

to coordinate speech during tool execution. Your understanding is correct. Key confirmations and guidance: • Use

session.say()

for immediate acknowledgement or progressive fillers. It’s designed for short, interruptible speech and does not create a full LLM turn. That makes it appropriate for quick “I’m checking that for you” messages and periodic status updates. • Do not use

generateReply()

for short filler speech.

generateReply()

creates a full LLM response, which can produce verbose or unintended content (like narrating the user’s intent) because the model treats it as a normal reply rather than an interruptible filler. • Inside tools, tools run concurrently with speech and other agent operations. You should coordinate playback with

context.wait_for_playout()

(or

ctx.wait_for_playout()

depending on how your tool receives the context) instead of awaiting the speech handle directly. That ensures proper synchronization without blocking the tool thread in ways that could cause unexpected behavior. • For long-running tools, pre-synthesized short hold audio is also a good option. Play it via

session.say()

(or as preloaded audio), allow interruptions, and cancel/interrupt it when the tool completes so the real reply can take over cleanly. About the JS SDK exposure: • You correctly called out the need to verify that the LiveKit JS SDK exposes

wait_for_playout()

on the tool context. If your tool

ctx

object includes

wait_for_playout()

, use it as described. If you don’t see it on your context object, tell me which SDK version and where the tool is executed (server-side tool vs client plugin) and I’ll help you find the exact API or a suitable alternative. If you’d like, I can: • Show a small JS code example that demonstrates using

session.say()

in a tool with

ctx.wait_for_playout()

coordination. • Review a short snippet of your current tool code and point out where to switch from

generateReply()

to

session.say()

and how to call

wait_for_playout()

. Which would you prefer?

we followed your recommendation and created a library of phrases to be said by the agent in english, any solution for multi language agents? because if convo is in spanish and then it says the phrase in english that breaks the experience

Great question — I’m responding to your concern about the agent playing pre-written English phrases during a Spanish conversation and how to make those phrases match the conversation language. Short answer: you can keep your phrase library approach and make it multilingual by either detecting the conversation language automatically and switching TTS/phrases, or letting the user/agent explicitly switch languages. Both approaches are supported. Two recommended approaches 1) Automatic language detection (seamless for users) • Detect the user language from STT. Deepgram supports automatic language detection with

language="multi"

, and you can read the detected language from the transcription results. • When the language is detected, update the agent’s TTS settings (voice/language) with

update_options(language=detected_language)

before calling

session.say()

so the filler phrase is spoken in the right language. • Keep your phrase library keyed by language (e.g.,

phrases['es'].acknowledgement

,

phrases['en'].acknowledgement

) and pick the phrase matching detected_language. • For coordination inside tools, continue to use

context.wait_for_playout()

after

session.say()

so playback and tool execution stay in sync. When to use this: best when you want a fully automatic, seamless experience and your users may switch languages mid-session. 2) Manual / function-based language switching (explicit user control) • Offer a small tool or function like

switch_language("es")

that updates STT and TTS options via

update_options()

and sets the active language for phrase lookup. • Keep per-language phrase dictionaries and call

session.say(phrases[currentLanguage].filler, { allowInterruptions: true })

. • This is useful when you want to respect a user’s explicit preference or when automatic detection might be ambiguous. Implementation tips and best practices • Maintain a phrase dictionary per language rather than translating phrases at runtime. This avoids awkward machine translations and preserves idiomatic phrasing. • Use short pre-synthesized hold audio (per language) for very frequent fillers to reduce latency; still allow interruption so the real reply interrupts the hold. • Make sure you update both STT and TTS in the same switch so recognition and playback match. • If you rely on automatic detection, add a small fallback: if detection confidence is low, ask the user (brief prompt) or default to the session’s configured language. • Continue using

session.say()

for short fillers (with allowInterruptions) rather than

generate_reply()

so you don’t create a full LLM turn that may produce unexpected text. If you want, I can: • Show a short JS (or Python) snippet demonstrating language detection -> update_options -> session.say() -> context.wait_for_playout(). • Review the shape of your phrase library and give a recommended structure for keys and fallbacks. Which would help you most — a code example in JS or Python, or a review of your current phrase structure? Sources: Build a multilingual voice agent that automatically switches languages | Li... | Deepgram STT | LiveKit Documentation | Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | ElevenLabs Change Language | LiveKit Documentation | Pipeline Translator Agent | LiveKit Documentation