how can I use agent.say() for realtime models? Is ...
# ask-ai
i
how can I use agent.say() for realtime models? Is there a way to force them to speak during tool calls ? @icy-policeman-64898
✅ 1
t
Thanks for reaching out, Akash. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — you’re asking about using
agent.say()
with realtime models and whether you can force them to speak during tool calls. Here’s how that works and what to do next. Short answer • You can use
session.say()
with realtime models, but the session must include a TTS plugin. • You can force speech during tool execution by calling
session.say()
(or
session.generate_reply()
) inside the tool. If you want the speech to play while the tool runs, call
say()
without awaiting it so it runs concurrently, and optionally interrupt it if a faster response arrives. What to configure • Use a realtime model for comprehension and add a separate TTS instance (half-cascade architecture). Example:
Copy code
session = AgentSession(
    llm=openai.realtime.RealtimeModel(modalities=["text"]),
    tts="cartesia/sonic-3"
  )
• If no TTS plugin is configured,
say()
won’t synthesize audio — use
generate_reply()
instead. Tool pattern example 1. Inside your function tool, trigger speech immediately:
Copy code
context.session.say("Let me check that for you.")
   result = await external_api_call(...)
   return result
1. Do not
await
the
say()
call if you want speech and the API call to run concurrently. 2. If the API returns early, interrupt or stop the speech handle per your audio customization needs. If you’d like, tell me: • Which SDK/language you’re using (Python/TypeScript/other). • Whether you already have a TTS plugin configured and which one. I’ll give a concrete code example tailored to your setup. Sources: Agent speech and audio | LiveKit Documentation | Function tool definition | LiveKit Documentation | The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon... | The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon... | Audio customization | LiveKit Documentation, with interactive elements
i
@icy-policeman-64898 I wanna configure xai's think fast 1.0 voice model but also be able to speak during tool calls using generate_reply or session.say() . put together the agent configuration for me
t
Thanks — I’ll build a concise agent configuration that uses XAI’s "think fast 1.0" TTS voice with a realtime model and demonstrates speaking during tool calls using both
session.say()
and
session.generate_reply()
. Which SDK do you want this in: Python or TypeScript? , with interactive elements
i
python
t
Thanks — I’ll give you a concise Python agent configuration that uses XAI's "think fast 1.0" TTS voice with a realtime model and shows speaking during tool calls using both
session.say()
and
session.generate_reply()
. Notes: • This config assumes your environment has the LiveKit Agents Python SDK installed and you have credentials for the realtime model and XAI TTS voice. • Replace placeholders (API keys, model IDs) with your values. Example Python agent configuration and tool usage:
Copy code
from livekit_agents import AgentSession, openai, function_tool, RunContext

# Configure the agent session with a realtime LLM and XAI TTS voice
session = AgentSession(
    llm=openai.realtime.RealtimeModel(
        model="openai.realtime.some-realtime-model",  # replace with your realtime model id
        modalities=["text"]
    ),
    tts="xai/think-fast-1.0"  # XAI Think Fast 1.0 TTS voice
)

# Example tool that speaks immediately while performing a long-running call
@function_tool()
async def lookup_order(context: RunContext, order_id: str):
    # Start spoken hold message without awaiting so it plays concurrently
    context.session.say("Let me look that up for you while I fetch the details.")
    # Simulate or call external async API
    result = await some_async_order_api.get(order_id)
    # Optionally generate a spoken summary after the result arrives
    await context.session.generate_reply(f"I found your order. Status: {result['status']}.")
    return result

# Example tool using generate_reply during the tool execution
@function_tool()
async def check_availability(context: RunContext, item_id: str):
    # Use generate_reply to synthesize a shorter spoken response during processing
    await context.session.generate_reply("Checking availability now.")
    avail = await inventory_api.check(item_id)
    return avail

# Register tools and start the agent session as your app requires
session.register_tool(lookup_order)
session.register_tool(check_availability)

# Start or run the session according to your app's lifecycle
# e.g., await session.start()
If you want, I can: • Replace placeholder model IDs with exact realtime model names if you share them. • Show a full runnable example including async server bootstrap and API-mocking for
some_async_order_api
., with interactive elements
i
can u use the xai plugin
t
Great — below is a concise Python agent configuration using the xAI plugin with the "think-fast-1.0" voice and examples showing speaking during tool calls with both
session.say()
and
session.generate_reply()
. Replace placeholders (API keys, model IDs, and external API calls) with your values.
Copy code
import os
import asyncio
from livekit.agents import AgentSession, function_tool, RunContext
from livekit.plugins import xai

# Ensure XAI API key is set in environment
# export XAI_API_KEY="your_xai_api_key"
# export LIVEKIT_API_KEY="your_livekit_api_key"   # if needed for LiveKit services

# Create an AgentSession using the xAI plugin realtime model and think-fast-1.0 voice
session = AgentSession(
    llm=xai.realtime.RealtimeModel(voice="think-fast-1.0"),  # xAI realtime model with Think Fast 1.0 voice
    # If you want separate TTS and realtime LLM, you can set tts="xai/think-fast-1.0" instead
)

# Example long-running tool that speaks a hold message using session.say()
@function_tool()
async def lookup_order(context: RunContext, order_id: str):
    # fire-and-forget spoken hold message so audio plays while we fetch data
    context.session.say("One moment — I’m pulling up your order details.")
    # simulate long-running external call
    await asyncio.sleep(2)  # replace with: result = await orders_api.get(order_id)
    result = {"order_id": order_id, "status": "shipped"}
    # speak a summary after the result arrives (await so reply completes before continuing)
    await context.session.generate_reply(f"I found order {order_id}. Status: {result['status']}.")
    return result

# Example tool that uses generate_reply() during processing and returns data
@function_tool()
async def check_availability(context: RunContext, item_id: str):
    # generate a short spoken message and receive a SpeechHandle
    handle = await context.session.generate_reply(f"Checking availability for item {item_id}.")
    # perform the check
    await asyncio.sleep(1)  # replace with: avail = await inventory_api.check(item_id)
    avail = {"item_id": item_id, "available": True}
    # optionally inspect handle.interrupted to decide next steps
    if getattr(handle, "interrupted", False):
        # user interrupted — optionally stop further speech or alter behavior
        return {"status": "interrupted"}
    # finalize with spoken confirmation
    await context.session.generate_reply(f"Item {item_id} is {'available' if avail['available'] else 'not available'}.")
    return avail

# Register tools with the session
session.register_tool(lookup_order)
session.register_tool(check_availability)

# Example entrypoint: start the session (adapt to your app runtime)
async def main():
    # Start or attach the session as your app requires
    await session.start()   # adjust per your app lifecycle (or await web server bootstrap)
    # Keep running for demo purposes
    try:
        while True:
            await asyncio.sleep(3600)
    finally:
        await session.stop()

if __name__ == "__main__":
    asyncio.run(main())
Notes and tips • Using
context.session.say(...)
without awaiting will play speech concurrently with your tool logic (good for hold messages). •
generate_reply()
returns a SpeechHandle you can inspect (e.g.,
handle.interrupted
) and you can await it when you want to ensure the spoken reply completes. • xAI plugin requires your XAI API key in the environment (e.g.,
XAI_API_KEY
). • If you prefer a half-cascade (realtime model for comprehension + separate xAI TTS), set
llm
to your realtime model and
tts="xai/think-fast-1.0"
in
AgentSession
. Want me to replace the sleep placeholders with a mocked API call or tailor this to a specific realtime LLM ID? Sources: Agent speech and audio | LiveKit Documentation | Text and transcriptions | LiveKit Documentation, with interactive elements