is it possible to have Realtime models (OpenAI Realtime or Gemini Live), output a recorded wav file as part of its output?

Thanks for reaching out, Sebastián. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

is not saving a WAV but the idea is given a tool call, that tool call has to play a specific wav file as it response, like a recording of a sound

Thanks — I’m responding to your question about having a tool call play a specific pre-recorded WAV (a recording/sound) as its response. Yes — you can play a pre-recorded WAV from a tool call. Load the WAV into audio frames once (e.g.,

audio_frames_from_file()

/

audioFramesFromFile()

), cache those frames, and pass them to

session.say()

as an

AsyncIterable[rtc.AudioFrame]

. You can include optional transcript text and manage playback with the returned

SpeechHandle

(interrupt/cancel if needed). This lets a tool play a static recording (hold message, sound effect, etc.) instead of synthesizing speech. Would you like a short Python or Node example showing preload + playback inside a tool? Sources: Audio customization | LiveKit Documentation | Agent speech and audio | LiveKit Documentation | Background audio | LiveKit Documentation, with interactive elements

but is that possible with Realtime models?

Short answer: Realtime models themselves do not directly play a pre-recorded WAV file. They produce/stream audio frames; they don’t natively return a file playback command. However, you can achieve exactly what you want inside the agent/tool flow that works with Realtime models: preload the WAV into audio frames (e.g.,

audio_frames_from_file()

/

audioFramesFromFile()

), then in the tool call use

session.say()

with those frames as an

AsyncIterable[rtc.AudioFrame]

. That plays the prerecorded WAV as the tool response (you can include text, and use the returned

SpeechHandle

to interrupt/cancel if needed). Would you like a short Python or Node sample of preload + tool playback? , with interactive elements

and is possible to combine Realtime model output + the recorded audio?

Short answer: Yes—you can combine Realtime model output with prerecorded audio, but not inside the model itself. Details: • Realtime models stream generated audio as audio frames; they do not natively play or return a WAV file or mix prerecorded audio for you. • Inside an agent/tool you can: ◦ Preload a WAV into audio frames (e.g.,

audio_frames_from_file()

/

audioFramesFromFile()

). ◦ Use

session.say()

with those frames (or with

text

+

audio

) to play the prerecorded sound as the tool response. ◦ Run

session.say()

concurrently with the Realtime model’s flow (capture the returned

SpeechHandle

so you can interrupt/cancel playback if the model replies early). • For true mixing/processing (volume, blending, concatenation) handle it on the agent side: either ◦ stream frames in sequence (play prerecorded frames then let model stream), or ◦ process/merge frames in a custom

realtime_audio_output_node

before sending to the client. Would you like a short Python or Node example showing preload + concurrent playback + interruption? Sources: Realtime models overview | LiveKit Documentation | Audio customization | LiveKit Documentation | Agent speech and audio | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation | Pipeline vs. Realtime - Which is the better Voice Agent Architecture? | Liv..., with interactive elements

if i use session.say inside a tool call, may i timeout the tool call?

Short answer: Yes—calling

session.say()

can cause a tool call to timeout if you await it. Don’t await

session.say()

if you need the tool to continue running without blocking; instead capture the returned

SpeechHandle

and let playback run concurrently. Details and guidance: • If you await

session.say()

, the tool will block until playback finishes (which can trigger tool-call timeouts). • Recommended pattern: a. Start playback without awaiting: • Python:

handle = context.session.say(... )

• Node:

const handle = context.session.say(...);

a. Continue the async work (e.g., call external APIs). b. If the tool finishes before playback should continue, call

handle.interrupt()

or

handle.cancel()

to stop playback early. • If you need playback to be part of the tool result (i.e., ensure it finishes before returning), then explicitly await it—but ensure your tool timeout is long enough. Would you like a brief Python or Node snippet showing the non-blocking pattern with interruption? , with interactive elements

and what alternative could we have to play long audios as result of tool call but making it part of the speaking state of the agent?

Acknowledging your question about alternatives to play long audios as part of the agent's speaking state. Short answer: Use session.say(...) with preloaded audio frames (and await it) to make long audio count as the agent speaking. Alternatives are BackgroundAudioPlayer (not part of speaking state) or custom mixing in a realtime_audio_output_node. If you want, I can: • Send a concise Python or Node example showing preload + session.say(await) to keep the agent in speaking state, or • Show a non-blocking pattern with SpeechHandle + interruption, or • Show how to use realtime_audio_output_node to merge/process streams. Which example would you like? Sources: Agent speech and audio | LiveKit Documentation | Background audio | LiveKit Documentation | Function tool definition | LiveKit Documentation | Audio customization | LiveKit Documentation, with interactive elements

im intereste in 1 and 3

Thanks — below are concise examples for both options you requested. 1) session.say with long audio (agent speaking state — await to make it part of speech) • Python

from livekit.agents import AgentSession
from livekit import rtc
from livekit.agents.utils.audio import audio_frames_from_file

async def play_long_audio_as_speech(session: AgentSession, path: str):
    # play and treat as agent speech; awaiting keeps tool blocked until finished
    await session.say(
        "Playing requested audio.",
        audio=audio_frames_from_file(path, sample_rate=24000, num_channels=1),
        allow_interruptions=True,  # optional: allow user to interrupt via VAD
    )

• Node

import { AgentSession } from "@livekit/agents";
import { audioFramesFromFile } from "@livekit/agents";

async function playLongAudioAsSpeech(session: AgentSession, path: string) {
  await session.say("Playing requested audio.", {
    audio: audioFramesFromFile(path, { sampleRate: 24000, numChannels: 1 }),
    allowInterruptions: true,
  });
}

Notes: • Awaiting the returned playout ensures the agent stays in the speaking state until completion. • Preload large files at startup into memory if you need low-latency playback. 2) Custom mixing / processing via realtime_audio_output_node (merge, chain, or alter streams before sending to client) • Concept: write a custom

realtime_audio_output_node

that consumes multiple sources (realtime model frames + prerecorded frames), mixes or sequences them, and yields final `AudioFrame`s. Plug this node into your agent’s audio pipeline. • Python (sketch)

from livekit import rtc

async def realtime_audio_output_node(model_audio_async_iter, prerecorded_iter):
    # model_audio_async_iter: AsyncIterable[rtc.AudioFrame] from realtime model
    # prerecorded_iter: AsyncIterable[rtc.AudioFrame] from preloaded WAV
    async for model_frame in model_audio_async_iter:
        # Example: yield model frames normally
        yield model_frame

    # After model finishes, play prerecorded frames
    async for frame in prerecorded_iter:
        yield frame

• More advanced mixing (overlay example)

import numpy as np

def mix_frames(frame_a: rtc.AudioFrame, frame_b: rtc.AudioFrame) -> rtc.AudioFrame:
    # convert to numpy, mix, clip, convert back — pseudocode; follow actual AudioFrame API
    a = frame_a.to_numpy()
    b = frame_b.to_numpy()
    mixed = a + b
    mixed = np.clip(mixed, -1.0, 1.0)
    return rtc.AudioFrame.from_numpy(mixed, sample_rate=frame_a.sample_rate, num_channels=frame_a.num_channels)

Implementation notes: • The node must respect frame formats (sample rate, channels) — convert/decimate as needed. • Use this node when you need sample-level control (volume, blending, seamless concatenation). • This runs agent-side before frames are sent to the client, so the client sees a single processed audio stream (agent speaking state is preserved if you tie this into the agent’s playout logic). If you want, I can: • Provide a full runnable Python example that preloads a WAV, uses session.say(await) in a tool, and caches frames, or • Expand the realtime_audio_output_node into a complete mixing example handling resampling and frame buffers. Which follow-up would you like? , with interactive elements

i don't want to inter-mix, but a very specific way to mix them

Thanks — quick clarification so I give the exact solution you need: Which specific mixing behavior do you want (pick one or describe): • Play prerecorded WAV immediately after the realtime model finishes speaking (strict sequence). • Play prerecorded WAV, then resume realtime model audio later (insert/sequence with resume). • At a specific cue in the model’s output replace the next N seconds with the prerecorded WAV (replace at cue). • Trigger the prerecorded WAV at a timestamp or external event while model audio pauses (event-driven swap). • Layer the prerecorded WAV under the model audio but keep model audio dominant (ducking). • Something else — please describe precisely. Also confirm: you want the combined result to be treated as the agent’s speaking state (interruptible by user VAD) and you plan to control playback from inside a tool call. , with interactive elements

sequentially not mix

Thanks — confirming: you want sequential playback (model audio first, then a prerecorded WAV) — not mixing. Below are concise patterns for that in both Python and Node. Python (await model playout, then play prerecorded):

from livekit.agents import AgentSession
from livekit import rtc
from livekit.agents.utils.audio import audio_frames_from_file

async def tool_sequential_play(context, model_audio_async_iter, wav_path: str):
    # 1) Stream model audio to client (this is your realtime model output async iterator)
    #    Ensure this iterator is routed into the agent playout pipeline so the client hears it.
    async for frame in model_audio_async_iter:
        await context.session.push_audio_frame(frame)  # pseudocode: send model frames

    # 2) After model finishes, play prerecorded WAV as agent speech and await completion
    await context.session.say(
        "Playing recorded audio now.",
        audio=audio_frames_from_file(wav_path, sample_rate=24000, num_channels=1),
        allow_interruptions=True,
    )

Node (same sequence):

import { audioFramesFromFile } from "@livekit/agents";

async function toolSequentialPlay(context, modelAudioAsyncIter, wavPath: string) {
  // 1) Send model frames sequentially
  for await (const frame of modelAudioAsyncIter) {
    await context.session.pushAudioFrame(frame); // pseudocode: send model frames
  }

  // 2) Then play prerecorded WAV and await playout
  await context.session.say("Playing recorded audio now.", {
    audio: audioFramesFromFile(wavPath, { sampleRate: 24000, numChannels: 1 }),
    allowInterruptions: true,
  });
}

Notes and key points: • The important part is to let the model audio finish (or detect its end) before calling

session.say()

for the WAV. • If model audio is produced via the agent’s normal playout pipeline, wait for that playout to complete (e.g., await its SpeechHandle or finish iterating model frames) before calling

session.say()

. • Use

allow_interruptions

and the returned SpeechHandle if you need to allow/capture user interruptions during the prerecorded playback. • Preload/ cache WAV frames (with

audio_frames_from_file

/

audioFramesFromFile

) for low latency if files are large. If you want, I can convert the pseudocode

push_audio_frame

into the exact API call for your setup (AgentSession streaming hook or realtime_audio_output_node). Which runtime (Python or Node) and how you receive model frames (agent playout, model iterator, or other) should I target? , with interactive elements

yes please, the main goal would be: a tool call fetches the file wav, we let realtime playout a message, when finish we replay our audio and ideally then we can [optionally] have the agent mention something else, otherwise go into listening

Thanks — I’ll show a clear, runnable pattern (both Python and Node) that does exactly what you described: Flow: 1. Tool call fetches/loads the WAV (cache frames). 2. Let the Realtime model playout its message to the client and await that playout. 3. Immediately after playout ends, call

session.say()

with the preloaded WAV and await it. 4. Optionally call

session.say()

with a short follow-up, or return so the agent goes back to listening. Python (async) — sketch with concrete APIs:

from livekit.agents import AgentSession, RunContext
from livekit.agents.utils.audio import audio_frames_from_file

# Preload (run at startup)
CACHE = {}
async def preload_wav(key: str, path: str):
    CACHE[key] = audio_frames_from_file(path, sample_rate=24000, num_channels=1)

# Tool that orchestrates sequential play
@function_tool()
async def play_seq(context: RunContext, wav_key: str, model_playout_handle):
    session: AgentSession = context.session

    # 1) Wait for realtime model playout to finish.
    # model_playout_handle should be a SpeechHandle or similar you get when starting model playout.
    # If you have an async iterator of model frames, await its completion or await the handle.
    if model_playout_handle is not None:
        await model_playout_handle  # ensure model audio finished

    # 2) Play fetched prerecorded WAV as agent speech (await to keep agent in speaking state)
    wav_frames = CACHE[wav_key]
    await session.say(
        "Playing the requested audio now.",        # optional transcript text
        audio=wav_frames,                          # AsyncIterable[AudioFrame]
        allow_interruptions=True,
    )

    # 3) Optional follow-up utterance, or simply return to go back to listening
    await session.say("That clip has finished. Anything else?", allow_interruptions=True)

    return {"status": "done"}

Node (async) — sketch with concrete APIs:

import { audioFramesFromFile } from "@livekit/agents";

// Preload (at startup)
const CACHE: Record<string, AsyncIterable<any>> = {};
function preloadWav(key: string, path: string) {
  CACHE[key] = audioFramesFromFile(path, { sampleRate: 24000, numChannels: 1 });
}

// Tool that orchestrates sequential play
export async function playSeq(context, wavKey: string, modelPlayoutHandle) {
  const session = context.session;

  // 1) Wait for realtime model playout to finish
  if (modelPlayoutHandle) {
    await modelPlayoutHandle; // SpeechHandle from model playout
  }

  // 2) Play prerecorded WAV as agent speech (await to keep speaking state)
  const wavFrames = CACHE[wavKey];
  await session.say("Playing the requested audio now.", {
    audio: wavFrames,
    allowInterruptions: true,
  });

  // 3) Optional follow-up
  await session.say("That clip has finished. Anything else?", { allowInterruptions: true });

  return { status: "done" };
}

Key integration points and notes: • How to detect model playout end: ◦ If your Realtime model playout is started via a function that returns a

SpeechHandle

, await that handle. ◦ If you have an async iterator of `AudioFrame`s from the model, finish iterating (or await the wrapper that pushes frames) before calling

session.say()

. • Always await the WAV

session.say()

if you want the WAV to be treated as agent speech (keeps agent in speaking state). • Use

allow_interruptions=True

(or equivalent) so user speech can stop playback; capture/handle interruption if needed. • Preload and cache large WAVs using

audio_frames_from_file

/

audioFramesFromFile

at startup to avoid fetch/decoding latency inside the tool. • If you fetch the WAV during the tool call, fetch & convert to frames first, then await model playout, then call

session.say()

with those frames. • If you want the optional follow-up to be skipped when user interrupts the WAV, check the returned

SpeechHandle

state or handle the interruption event before issuing the follow-up. If you want, I can: • Convert the sketches into a complete runnable example tied to how you currently start model playout (provide the code that starts model audio so I can show exact handle usage), or • Show a variant that fetches the WAV from a remote URL inside the tool and caches it before playback. Which follow-up would you like? , with interactive elements

since audio can be long, i don't want to get a tool call timeout or similar, also i need to get out of thinking since we play a thinking sound client side

Thanks — I’ll address both goals: avoid tool-call timeouts for long audio playback, and exit the “thinking” state (client-side thinking sound) before the long prerecorded file plays. High-level approach (what to do) • Keep the long playback off the tool’s critical path: start it from the agent/session but don’t block the tool call on the full playout. • Use a two-phase flow: a. Tool finishes quickly after coordinating playback: tell the client to stop the thinking sound and start agent playout, then return so the tool does not time out. b. Playback and any follow-up speech are handled via the agent’s speaking playout (session.say) and SpeechHandle management (run asynchronously), or by awaiting playout only when you explicitly want the tool to wait. • Preload or stream the WAV frames before playout to avoid runtime decoding delays. Concrete pattern (sequence you want) 1. Client plays short thinking audio locally while tool is running. 2. Tool fetches/loads WAV frames (or ensures they’re cached). 3. Tool triggers Realtime model playout (or streams model frames) and signals client to stop thinking sound and switch to agent playout. 4. Tool returns quickly (no await on long playout) to avoid timeouts. 5. Agent/session continues playout server-side: ◦ Play Realtime model audio (if applicable). ◦ After model playout completes, start

session.say()

with prerecorded WAV and await it if you want to know when finished, or don’t await if you want to remain non-blocking. 1. Optionally issue a short follow-up

session.say()

after WAV finishes (await or run as separate task). Key APIs & behaviors to use • session.say(audio=...) with preloaded AsyncIterable[AudioFrame] to play prerecorded WAV as agent speech. • session.say(...) returns a SpeechHandle — use it to interrupt/cancel or to await completion. • Don’t await long SpeechHandle inside the tool if you must avoid tool timeouts; instead manage playback asynchronously after tool returns. • Preload via audio_frames_from_file / audioFramesFromFile at startup or during a non-critical path. Minimal Python sketch (non-blocking tool, agent continues playback)

# run at startup
CACHE["clip"] = audio_frames_from_file("clip.wav", sample_rate=24000, num_channels=1)

@function_tool()
async def orchestrate_play(context: RunContext, wav_key: str, model_playout_handle=None):
    session = context.session

    # Ensure WAV is available (decoded frames cached)
    wav_frames = CACHE[wav_key]

    # 1) If model playout is started elsewhere and you have a handle, wait for it to finish *only if* you want server-side sequencing.
    #    To avoid tool timeout, do NOT await a long playout here. Instead, start a background task.
    async def background_sequence():
        # If you have a model SpeechHandle, await it here to ensure model audio finishes before WAV
        if model_playout_handle:
            await model_playout_handle

        # 2) Play prerecorded WAV as agent speech (awaiting here is fine because this is background)
        wav_handle = session.say("Playing clip now.", audio=wav_frames, allow_interruptions=True)
        await wav_handle

        # 3) Optional follow-up utterance
        await session.say("That clip finished. Anything else?", allow_interruptions=True)

    # 3) Signal client to stop thinking sound and switch to agent playout (send event in tool response)
    # Return quickly to avoid timeout; spawn background task to perform actual long playout sequence.
    context.loop.create_task(background_sequence())  # or asyncio.create_task(...)
    return {"status": "started", "action": "stop_thinking_and_listen_for_agent"}

Minimal Node sketch (same idea)

// preload at startup
CACHE["clip"] = audioFramesFromFile("clip.wav", { sampleRate: 24000, numChannels: 1 });

export async function orchestratePlay(context, wavKey, modelPlayoutHandle) {
  const session = context.session;
  const wavFrames = CACHE[wavKey];

  const backgroundSequence = async () => {
    if (modelPlayoutHandle) {
      await modelPlayoutHandle;
    }
    const wavHandle = session.say("Playing clip now.", { audio: wavFrames, allowInterruptions: true });
    await wavHandle;
    await session.say("That clip finished. Anything else?", { allowInterruptions: true });
  };

  // start background, return immediately so tool doesn't time out
  void backgroundSequence();
  return { status: "started", action: "stop_thinking_and_listen_for_agent" };
}

Client-side coordination • While tool runs: play local short thinking sound. • When tool response arrives: stop thinking sound and switch to agent audio playout (so user hears agent audio coming from server). • Use a small event field in tool response (e.g., action: "stop_thinking_and_listen_for_agent") so client knows to stop local audio and allow server audio through. Notes & best practices • Preload WAV frames to avoid decoding time inside the background task. • If you need guaranteed sequencing (model audio → WAV → follow-up) do sequencing in the background task by awaiting SpeechHandles in order. • Use SpeechHandle interruption if user speaks during the WAV and you want to cancel follow-ups. • Keep tool execution fast (fetch/cache then spawn background task) to prevent platform timeouts. If you want, I can: • Produce a complete runnable example tied to your exact runtime (how you currently start model playout and signal the client). Which runtime and client signaling mechanism are you using?, with interactive elements

but for session.say, you need a different model aside from realtime right?

Short answer: No —

session.say()

does not require a different model. You can use it alongside a Realtime model. Details: • Realtime models stream/generated audio frames; they don’t natively play prerecorded WAVs. • Use the agent/session APIs together: run your Realtime model playout, then call

session.say(audio=...)

(with preloaded WAV frames) to play the prerecorded clip as agent speech.

session.say()

simply plays audio frames (or synthesizes TTS) — it’s independent of which model produced prior audio. • For sequencing without timeouts: start the model playout, capture/await its SpeechHandle (or await the model frame iterator) in a background task, then call

session.say()

for the WAV and optionally follow-up speech — spawn that background task from the tool so the tool returns immediately. If you want, I can give a minimal runnable example that shows exactly how to start model playout, capture its handle, and then call

session.say()

for the WAV in a background task (Python or Node). Which language? , with interactive elements

per your docs, you need to do so

Thanks — I hear the confusion. You’re right to double-check the docs; here’s the clear answer. Short answer: You do NOT need a different model to use

session.say()

.

session.say()

is an agent/session API that plays audio frames (or synthesizes TTS) and can be used alongside a Realtime model — sequencing and playback are handled by your agent/session code, not by switching models. Key points • Realtime models stream generated audio frames; they don’t themselves play prerecorded WAVs or issue playback commands. •

session.say(audio=...)

accepts an AsyncIterable of

AudioFrame

(preloaded WAV frames) and will play that audio as agent speech — independent of which model produced prior audio. • To avoid tool timeouts and exit the client “thinking” state: ◦ Have the tool preload/fetch the WAV, spawn a background task that awaits the realtime playout SpeechHandle, then calls

session.say()

for the WAV and any follow-up. ◦ Return from the tool immediately with a small signal (e.g., action: "stop_thinking_and_listen_for_agent") so the client stops the local thinking sound and switches to server playout. •

session.say()

returns a

SpeechHandle

you can await or keep in background to manage interruption/cancellation. If you want, I’ll produce a complete runnable example tailored to your stack (Python or Node) that: • starts realtime playout, • returns from the tool immediately (so no timeout), • runs the sequential background task that awaits model playout, plays the WAV via

session.say()

, and optionally does a follow-up. Which language (Python or Node) and how do you currently start realtime playout (do you get a SpeechHandle or an async iterator)? , with interactive elements