incalculable-fireman-8935
05/10/2026, 8:26 PMtall-airline-16756
05/10/2026, 8:26 PMtall-airline-16756
05/10/2026, 8:26 PMagent.say() with realtime models and whether you can force them to speak during tool calls. Here’s how that works and what to do next.
Short answer
• You can use session.say() with realtime models, but the session must include a TTS plugin.
• You can force speech during tool execution by calling session.say() (or session.generate_reply()) inside the tool. If you want the speech to play while the tool runs, call say() without awaiting it so it runs concurrently, and optionally interrupt it if a faster response arrives.
What to configure
• Use a realtime model for comprehension and add a separate TTS instance (half-cascade architecture). Example:
session = AgentSession(
llm=openai.realtime.RealtimeModel(modalities=["text"]),
tts="cartesia/sonic-3"
)
• If no TTS plugin is configured, say() won’t synthesize audio — use generate_reply() instead.
Tool pattern example
1. Inside your function tool, trigger speech immediately:
context.session.say("Let me check that for you.")
result = await external_api_call(...)
return result
1. Do not await the say() call if you want speech and the API call to run concurrently.
2. If the API returns early, interrupt or stop the speech handle per your audio customization needs.
If you’d like, tell me:
• Which SDK/language you’re using (Python/TypeScript/other).
• Whether you already have a TTS plugin configured and which one.
I’ll give a concrete code example tailored to your setup.
Sources: Agent speech and audio | LiveKit Documentation | Function tool definition | LiveKit Documentation | The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon... | The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon... | Audio customization | LiveKit Documentation, with interactive elementsincalculable-fireman-8935
05/10/2026, 8:53 PMtall-airline-16756
05/10/2026, 8:53 PMsession.say() and session.generate_reply(). Which SDK do you want this in: Python or TypeScript?
, with interactive elementsincalculable-fireman-8935
05/10/2026, 8:53 PMtall-airline-16756
05/10/2026, 8:54 PMsession.say() and session.generate_reply().
Notes:
• This config assumes your environment has the LiveKit Agents Python SDK installed and you have credentials for the realtime model and XAI TTS voice.
• Replace placeholders (API keys, model IDs) with your values.
Example Python agent configuration and tool usage:
from livekit_agents import AgentSession, openai, function_tool, RunContext
# Configure the agent session with a realtime LLM and XAI TTS voice
session = AgentSession(
llm=openai.realtime.RealtimeModel(
model="openai.realtime.some-realtime-model", # replace with your realtime model id
modalities=["text"]
),
tts="xai/think-fast-1.0" # XAI Think Fast 1.0 TTS voice
)
# Example tool that speaks immediately while performing a long-running call
@function_tool()
async def lookup_order(context: RunContext, order_id: str):
# Start spoken hold message without awaiting so it plays concurrently
context.session.say("Let me look that up for you while I fetch the details.")
# Simulate or call external async API
result = await some_async_order_api.get(order_id)
# Optionally generate a spoken summary after the result arrives
await context.session.generate_reply(f"I found your order. Status: {result['status']}.")
return result
# Example tool using generate_reply during the tool execution
@function_tool()
async def check_availability(context: RunContext, item_id: str):
# Use generate_reply to synthesize a shorter spoken response during processing
await context.session.generate_reply("Checking availability now.")
avail = await inventory_api.check(item_id)
return avail
# Register tools and start the agent session as your app requires
session.register_tool(lookup_order)
session.register_tool(check_availability)
# Start or run the session according to your app's lifecycle
# e.g., await session.start()
If you want, I can:
• Replace placeholder model IDs with exact realtime model names if you share them.
• Show a full runnable example including async server bootstrap and API-mocking for some_async_order_api., with interactive elementsincalculable-fireman-8935
05/10/2026, 8:58 PMtall-airline-16756
05/10/2026, 8:58 PMsession.say() and session.generate_reply().
Replace placeholders (API keys, model IDs, and external API calls) with your values.
import os
import asyncio
from livekit.agents import AgentSession, function_tool, RunContext
from livekit.plugins import xai
# Ensure XAI API key is set in environment
# export XAI_API_KEY="your_xai_api_key"
# export LIVEKIT_API_KEY="your_livekit_api_key" # if needed for LiveKit services
# Create an AgentSession using the xAI plugin realtime model and think-fast-1.0 voice
session = AgentSession(
llm=xai.realtime.RealtimeModel(voice="think-fast-1.0"), # xAI realtime model with Think Fast 1.0 voice
# If you want separate TTS and realtime LLM, you can set tts="xai/think-fast-1.0" instead
)
# Example long-running tool that speaks a hold message using session.say()
@function_tool()
async def lookup_order(context: RunContext, order_id: str):
# fire-and-forget spoken hold message so audio plays while we fetch data
context.session.say("One moment — I’m pulling up your order details.")
# simulate long-running external call
await asyncio.sleep(2) # replace with: result = await orders_api.get(order_id)
result = {"order_id": order_id, "status": "shipped"}
# speak a summary after the result arrives (await so reply completes before continuing)
await context.session.generate_reply(f"I found order {order_id}. Status: {result['status']}.")
return result
# Example tool that uses generate_reply() during processing and returns data
@function_tool()
async def check_availability(context: RunContext, item_id: str):
# generate a short spoken message and receive a SpeechHandle
handle = await context.session.generate_reply(f"Checking availability for item {item_id}.")
# perform the check
await asyncio.sleep(1) # replace with: avail = await inventory_api.check(item_id)
avail = {"item_id": item_id, "available": True}
# optionally inspect handle.interrupted to decide next steps
if getattr(handle, "interrupted", False):
# user interrupted — optionally stop further speech or alter behavior
return {"status": "interrupted"}
# finalize with spoken confirmation
await context.session.generate_reply(f"Item {item_id} is {'available' if avail['available'] else 'not available'}.")
return avail
# Register tools with the session
session.register_tool(lookup_order)
session.register_tool(check_availability)
# Example entrypoint: start the session (adapt to your app runtime)
async def main():
# Start or attach the session as your app requires
await session.start() # adjust per your app lifecycle (or await web server bootstrap)
# Keep running for demo purposes
try:
while True:
await asyncio.sleep(3600)
finally:
await session.stop()
if __name__ == "__main__":
asyncio.run(main())
Notes and tips
• Using context.session.say(...) without awaiting will play speech concurrently with your tool logic (good for hold messages).
• generate_reply() returns a SpeechHandle you can inspect (e.g., handle.interrupted) and you can await it when you want to ensure the spoken reply completes.
• xAI plugin requires your XAI API key in the environment (e.g., XAI_API_KEY).
• If you prefer a half-cascade (realtime model for comprehension + separate xAI TTS), set llm to your realtime model and tts="xai/think-fast-1.0" in AgentSession.
Want me to replace the sleep placeholders with a mocked API call or tailor this to a specific realtime LLM ID?
Sources: Agent speech and audio | LiveKit Documentation | Text and transcriptions | LiveKit Documentation, with interactive elements