from

agents/src/voice/speech_handle.ts

/**
   * Waits for the entire assistant turn to complete playback.
   *
   * This method waits until the assistant has fully finished speaking,
   * including any finalization steps beyond initial response generation.
   * This is appropriate to call when you want to ensure the speech output
   * has entirely played out, including any tool calls and response follow-ups.
   */
  async waitForPlayout(): Promise<void> {
    const store = functionCallStorage.getStore();
    if (store && store?.functionCall) {
      throw new Error(
        `Cannot call 'SpeechHandle.waitForPlayout()' from inside the function tool '${store.functionCall.name}'. ` +
          'This creates a circular wait: the speech handle is waiting for the function tool to complete, ' +
          'while the function tool is simultaneously waiting for the speech handle.\n' +
          "To wait for the assistant's spoken response prior to running this tool, use RunContext.wait_for_playout() instead.",
      );
    }
    await this.doneFut.await;
  }

If that's the case, how can I play a prompt within a tool and wait for playout?

Thanks for reaching out, Ahron. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

context.wait_for_playout()

only waits for playouts called prior to the tool call

Thanks — you’re right to call that out. You pointed out that

context.wait_for_playout()

only waits for playouts that were started before the tool ran. That is correct. What to do instead • If you need to start speech from inside the function tool and wait for that specific speech to finish, start the speech inside the tool and await the returned SpeechHandle’s

wait_for_playout()

method. That handle tracks the playback that it started, so awaiting it won’t trigger the circular-wait error. Minimal TypeScript example (inside your tool)

async function myTool(context: RunContext) {
  // start speech during the tool execution
  const handle = context.session.say("Please hold on while I run the task.");

  // wait for that speech to finish
  await handle.waitForPlayout();

  // handle interruption if needed
  if (handle.interrupted) {
    console.log("Speech was interrupted");
  } else {
    console.log("Speech finished normally");
  }

  // continue tool logic
}

Why this works •

context.wait_for_playout()

only awaits playouts that were in-flight or queued before the tool began. • The SpeechHandle returned by

session.say()

/

session.generate_reply()

represents the playback you started inside the tool; awaiting that handle waits only for that playback and avoids the circular wait detection. If you want non-blocking behavior • Add a done callback to the handle instead of awaiting it, so the tool can continue while you get notified when playback finishes. If you’d like, tell me: • the language/runtime you’re using (TypeScript or Python) and whether you use

say()

or

generate_reply()

, and I’ll give a concrete snippet matching your code and any interruption-handling patterns you prefer. Sources: Agent speech and audio | LiveKit Documentation | How to detect when an agent has finished speaking | LiveKit

the source code at agents/src/voice/speech_handle.ts doesn't allow a speech handle await for playout

Thanks — I hear your concern: the source in

agents/src/voice/speech_handle.ts

intentionally prevents using

SpeechHandle.waitForPlayout()

from inside a function tool because that creates a circular wait. Let’s walk through what that check means, why it exists, and practical patterns you can use to play audio inside a tool and wait for its completion despite the restriction. What the source check does • The code checks for a function-tool execution context and throws if

SpeechHandle.waitForPlayout()

is called from inside it. That prevents the speech system and the tool runtime from waiting on each other simultaneously (a circular wait). Why it’s enforced • If the tool is waiting on the speech system while the speech system is (directly or indirectly) waiting for the tool to finish, both will hang or cause deadlocks. The guard avoids that class of deadlocks. If you need to play speech inside a tool and wait for that speech to finish • You cannot call the top-level

SpeechHandle.waitForPlayout()

method if that method triggers the same circular-check path. Instead, use one of these safe patterns depending on what the runtime exposes and your needs: Option A — Await the handle’s internal completion future directly • When you start speech inside the tool (e.g.,

context.session.say()

or

context.session.generate_reply()

), capture the returned handle and await its internal completion promise/future rather than calling the guarded

SpeechHandle.waitForPlayout()

wrapper. • In other words, await the handle’s underlying done future (often named

doneFut

,

done

, or similar) that represents the actual playback completion. This avoids the external circular-check because you’re waiting on the specific playback instance created by the tool. TypeScript-style sketch:

async function myTool(context: RunContext) {
  const handle = context.session.say("Playing inside tool");

  // Wait for the handle's internal completion future directly
  await handle.doneFut.await();   // or await handle.doneFut; depending on API shape

  if (handle.interrupted) {
    // handle interruption
  }
  // continue tool logic
}

Option B — Use callbacks instead of awaiting • Add a completion callback on the handle so your tool can continue non-blocking and react when playout finishes.

const handle = context.session.say("Playing...");
handle.add_done_callback(() => {
  // run post-playout work
});

Option C — Restructure control flow: start speech before entering the tool, then use

context.wait_for_playout()

• If the speech can be initiated prior to invoking the tool, start it outside the tool and then let the tool call

context.wait_for_playout()

— that method only waits for playouts started before the tool began, avoiding circular checks. • This is only possible when you can reorder operations so the speech starts earlier. How to choose • If you must play the speech inside the tool and block until it finishes: Option A (await the handle's internal completion future) is the direct solution. • If you want the tool to continue work and be notified when playback finishes: Option B (callbacks). • If you can start the speech before the tool runs: Option C (use

context.wait_for_playout()

). If you’re seeing a specific error or the handle does not expose a `doneFut`/internal future that you can await, please share: • The exact runtime language (TypeScript or Python) and the snippet you used to start the speech. • The exact error message or stack trace you get when awaiting the handle. With that I can give a concrete snippet matching the actual API names in your runtime (e.g., whether the internal future is

doneFut

,

done

,

awaitDone()

, or something else) and show the correct line to await or how to attach a callback.

I want to always play a prompt right before the tool execution. how can I accomplish it without calling the generateReply in the tool itself?

Thanks — you want to always play a prompt immediately before a tool runs, but without calling `generate_reply()`/`say()` from inside the tool. That’s doable by initiating the speech before the tool invocation and letting the tool wait for that pre-started playout. Below are clear patterns and short examples you can use. Key idea • Start the prompt outside the tool (e.g., in your agent logic or a pre-tool step) using

session.generate_reply()

or

session.say()

. • Inside the tool, call

context.wait_for_playout()

to wait for that pre-started prompt to finish. This avoids circular waits and keeps speech sequencing predictable. When to use each pattern • If you can control agent flow: start the prompt in the agent controller (before the LLM decides to call the tool). • If you need the LLM to still decide to call the tool: ensure the LLM receives the prompt first; when it invokes the tool, the tool’s

context.wait_for_playout()

will wait for that prompt to finish. • If you need to guarantee the prompt always plays immediately before any tool: play it from a wrapper or a pre-tool hook that runs before each tool invocation. Minimal TypeScript example (agent initiates speech, tool waits)

// Agent logic - start prompt before LLM/tool choice
this.session.generateReply({ instructions: "Please wait—starting the action now." });

// Later the LLM chooses to call the tool. Inside the tool:
async function myTool(context: RunContext) {
  // Waits for the prompt that was started above
  await context.waitForPlayout();

  // Now safe to perform the tool work
  return await doToolWork();
}

Minimal Python example

# Agent code: play prompt before tool execution
self.session.generate_reply(instructions="Please wait while I run this check.")

# Inside the function tool
@function_tool()
async def my_tool(self, context: RunContext, ...):
    await context.wait_for_playout()   # waits for the above prompt
    result = await self._do_work(...)
    return result

Making this automatic for every tool • Option A (preferred): Add a small wrapper/hook that runs before any tool is invoked: ◦ Wrapper starts the prompt, then forwards execution to the tool. The tool can still call

context.wait_for_playout()

or the wrapper can wait and then call the tool. • Option B: If your agent runtime exposes a per-tool pre-hook or middleware, use it to call

session.generate_reply()

and then invoke the tool. The tool can assume the prompt has been started. • Option C (explicit): Always call the prompt from your agent logic right before the user action that will cause the LLM to invoke a tool. Notes and gotchas • Do not call

SpeechHandle.waitForPlayout()

from inside the tool if that method triggers the circular-check path; use

context.wait_for_playout()

to wait for pre-started speech instead. •

context.wait_for_playout()

only waits for speech that was started before the tool began. That’s why you must start the prompt before the tool is invoked. • If you need to start the prompt and immediately block until it finishes before invoking the tool, start the prompt then await its handle outside the tool, then call the tool afterward. But that changes timing (tool only runs after speech fully completes). • If you want non-blocking behavior inside the tool, attach a done callback to the speech handle instead of awaiting. If you’d like, tell me: • The language/runtime (TypeScript or Python) and whether you want this enforced for all tools automatically (a wrapper) or done manually per-tool, and I’ll provide a concrete wrapper or hook example that fits your codebase. Sources: Agent speech and audio | LiveKit Documentation | How to detect when an agent has finished speaking | LiveKit

I don't get it. The agent decides when to invoke the tool. How can I ensure the prompt is being played?

You’re right to be confused — the agent (LLM) decides when to call a tool, so you can’t reliably play a prompt “right before” the tool from outside unless you control the step that triggers the tool. Below are concrete, practical patterns you can use so the prompt always plays before the tool runs, plus which one to pick based on how much control you have. Quick summary • You must ensure speech is initiated in a step that always occurs before the LLM is allowed to call the real tool. • Then, inside the tool, call

context.wait_for_playout()

so the tool blocks until that pre-started speech finishes. • The common ways to guarantee that pre-step are: (A) a wrapper/pre-tool that plays the prompt then delegates to the tool, (B) an agent-level orchestration step that always plays the prompt before exposing the tool to the LLM, or (C) a two-step LLM interaction where the model is forced to acknowledge the prompt before deciding to call the tool. Pick one of these patterns: 1) Wrapper tool (most reliable, minimal change to LLM behavior) • Create a light wrapper tool the LLM calls instead of the real tool. • Wrapper plays the prompt (outside the real tool), then invokes the real tool. • Real tool uses

context.wait_for_playout()

to wait for the wrapper-started speech. Why this works: The wrapper is invoked first by the LLM; because the wrapper initiates speech before delegating, the real tool will always see that pre-started speech. Example (conceptual): 1. LLM calls

confirm_and_run_tool(...)

(the wrapper). 2. Wrapper does

session.generate_reply("Playing prompt...")

. 3. Wrapper calls the real tool (or directly awaits the real tool function). 4. Real tool runs

await context.wait_for_playout()

and then performs work. 2) Agent orchestration / pre-step before exposing the tool • In your agent controller (the code that manages turns), always play the prompt whenever you know a tool call might follow. • Then let the LLM run and decide whether to call the tool; when it does, the tool calls

context.wait_for_playout()

to wait for that pre-started speech. Why this works: You control the point where the LLM is given the option to call the tool, and you ensure the prompt is started just before that decision point. 3) Two-step LLM flow (prompt as an explicit confirmation) • Force a two-message interaction: first, send the LLM a reply that contains the prompt text and await the LLM’s next message (or an explicit confirmation). The LLM’s next message then may call the tool. • Because the prompt was sent before the tool call, the tool can use

context.wait_for_playout()

. Why this works: By requiring the LLM to produce a follow-up (or to emit a special token/decision), you ensure ordering without modifying tool definitions. 4) If you must avoid wrappers and orchestration: use a short pre-tool “prompt” tool • Provide the LLM with a small built-in tool that only plays the prompt (no other logic). The LLM must call that tool immediately before the main tool (you can encourage this by tool descriptions or instructions). • Then require the main tool to call

context.wait_for_playout()

as usual. Risk: relies on the LLM following the intended sequence; less reliable than the wrapper. Which to choose • If you can change how the LLM is called (you control the orchestration layer): use agent orchestration (option 2). • If you can change the tool list the LLM sees but want a bulletproof guarantee: use a wrapper tool that always runs first (option 1). • If you can modify conversation flow and accept a two-step exchange: use the two-step flow (option 3). • If you can only add a tool but not guarantee LLM behavior: add a prompt-only tool, but note it’s less reliable (option 4). Concrete next step I can provide • If you tell me your runtime (TypeScript or Python) and whether you prefer a wrapper or agent-orchestration pattern, I’ll give a focused code snippet you can drop into your codebase that: ◦ starts the prompt before the real work, and ◦ demonstrates the tool calling

context.wait_for_playout()

to block until the prompt finishes. Would you like a TypeScript wrapper example or a Python agent-orchestration snippet?

give me an example of a wrapper tool using js sdk

Thanks — below is a concrete JavaScript (Node) example showing a wrapper tool that guarantees a prompt is played before the real tool runs. The wrapper starts the speech outside the real tool, then delegates to the real tool; the real tool uses

ctx.waitForPlayout()

to block until that pre-started prompt completes. Why this pattern works • The LLM calls the wrapper first. The wrapper starts the prompt (via

session.generateReply()

/

session.say()

), then invokes the real tool. • The real tool calls

ctx.waitForPlayout()

to wait for the prompt that was started by the wrapper. Because the prompt was started before the tool began,

ctx.waitForPlayout()

will wait for it without causing a circular wait. Example (JavaScript SDK)

// Assuming you have the LiveKit agent/voice SDK imported and setup
// This example is conceptual; adapt to your project's module/layout.

const { voice, llm } = require('livekit-agents'); // adjust import to your setup

// Real tool: actual work happens here. It expects the prompt to have been started already.
const realTool = llm.tool({
  name: 'real_tool',
  description: 'Performs the real operation after confirmation prompt.',
  parameters: llm.z.object({
    input: llm.z.string(),
  }),
  execute: async ({ input }, { ctx }) => {
    // Wait for the prompt that the wrapper started to finish playing.
    await ctx.waitForPlayout();

    // Now perform the real work
    // ... your operation logic
    const result = `Processed: ${input}`;
    return { result };
  },
});

// Wrapper tool: starts the prompt, then delegates to the real tool.
const wrapperTool = llm.tool({
  name: 'confirm_and_run_tool',
  description:
    'Plays a confirmation prompt, then runs the real tool. Use this to ensure prompt plays before execution.',
  parameters: llm.z.object({
    input: llm.z.string(),
  }),
  execute: async ({ input }, { session, tools }) => {
    // 1) Start speech BEFORE delegating to the real tool
    // Use generateReply to create the spoken prompt (alternatively, use session.say()).
    session.generateReply({
      instructions: 'Please hold on — I will run the requested operation now.',
      // You can set allowInterruptions to true/false as needed
      allowInterruptions: false,
    });

    // 2) Delegate to the real tool. The LLM may not automatically switch context,
    // so we can call the tool programmatically here to ensure flow,
    // or return instructions so the LLM invokes the real tool next.
    //
    // Option A: Programmatically call the real tool from the wrapper and return its result:
    const toolResult = await tools.real_tool.execute({ input }, { ctx: /* pass appropriate RunContext if needed */ });

    // Option B: If you want the LLM itself to call the real tool (and keep wrapper minimal),
    // simply return a structured signal or instruction for the LLM to call `real_tool`.
    //
    // Here we used Option A to guarantee immediate delegation and return the final result.
    return toolResult;
  },
});

// Agent setup: expose wrapperTool (and realTool if desired)
const agent = new voice.Agent({
  instructions: 'You are a helpful assistant.',
  tools: {
    confirm_and_run_tool: wrapperTool,
    real_tool: realTool, // optional: exposing real tool directly is fine but LLM should call wrapper
  },
});

// When the LLM chooses to invoke the wrapper tool, the wrapper will play the prompt
// before delegating to the real tool. The real tool uses ctx.waitForPlayout() to pause
// until the prompt finishes playing.

Notes and options • Delegation style: ◦ Option A (programmatic delegation inside wrapper): wrapper calls

tools.real_tool.execute(...)

directly and returns the result. This is the most reliable, because it doesn’t rely on the LLM to call the real tool next. ◦ Option B (LLM-mediated delegation): wrapper returns a response that causes the LLM to call the real tool next. This relies on the LLM following sequence, but still works if you instruct it clearly. • Use

session.generateReply()

or

session.say()

in the wrapper. Both start the speech that

ctx.waitForPlayout()

will wait for in the real tool. • Ensure

allowInterruptions

and other speech settings match desired behavior (e.g., disallow interruptions if you need the prompt to finish). • If you programmatically call

tools.real_tool.execute(...)

from the wrapper, you may need to pass a proper

RunContext

or adapt to how your SDK wires contexts. If you cannot pass the same

ctx

object, an alternative is to have the wrapper return a clear instruction so the LLM invokes the real tool next (Option B). If you want, I can: • Convert this to a fully runnable TypeScript example matching your project layout. • Show the Option B flow where the wrapper returns instructions and the LLM calls the real tool next. Which would you prefer?

how can I call the real tool from within the wrapper?

You're on the right track — you can call the real tool from inside a wrapper by keeping a reference to the real tool's

execute

function (or using the agent's

tools

map) and invoking it after you start the prompt. Below is a concise, practical JavaScript (Node) example using the LiveKit JS SDK style that shows both approaches and handles waiting for the prompt playout. Notes up front • Start the speech in the wrapper (via

session.generateReply()

or

session.say()

). • Then delegate to the real tool by calling its

execute()

function (programmatic delegation) so you don't rely on the LLM to call it next. • Ensure the real tool uses

ctx.waitForPlayout()

if it needs to wait for the pre-started prompt to complete. Example (wrapper calls real tool programmatically)

import { voice, llm } from '@livekit/agents';
import { z } from 'zod';

// Real tool (actual work)
const realTool = llm.tool({
  name: 'real_tool',
  description: 'Performs the real operation after confirmation prompt.',
  parameters: z.object({ input: z.string() }),
  execute: async ({ input }, { ctx }) => {
    // If the prompt was started before this tool ran, this waits for it to finish
    await ctx.waitForPlayout();

    // Real tool logic
    const result = `Processed: ${input}`;
    return { result };
  },
});

// Wrapper tool: starts the prompt, then calls realTool.execute()
const wrapperTool = llm.tool({
  name: 'confirm_and_run_tool',
  description:
    'Plays a confirmation prompt, then runs the real tool. This wrapper delegates to the real tool directly.',
  parameters: z.object({ input: z.string() }),
  execute: async ({ input }, { session }) => {
    // 1) Start the prompt before delegating
    session.generateReply({
      instructions: `Please hold on — running the operation for "${input}" now.`,
      allowInterruptions: false,
    });

    // 2) Programmatically call the real tool's execute function and return result
    // Note: pass the same context if you have it; here we rely on the SDK wiring
    const result = await realTool.execute({ input }, { ctx: /* ctx not available here */ });

    // If your wrapper's execute signature gives you ctx, pass it through:
    // const result = await realTool.execute({ input }, { ctx });

    return result;
  },
});

// Agent where wrapper is exposed (expose wrapper in tools so LLM calls it)
const agent = new voice.Agent({
  instructions: 'You are a helpful assistant.',
  tools: {
    confirm_and_run_tool: wrapperTool,
    // real_tool can be omitted from the public tools if you want to force wrapper usage
  },
});

Important implementation details • Context passing: many SDKs provide

ctx

(RunContext) in the tool

execute

args. If your wrapper receives

{ ctx }

, pass that same

{ ctx }

into

realTool.execute(...)

so the real tool has the correct execution context (and

ctx.waitForPlayout()

will work). ◦ If your wrapper's signature only receives

{ session }

or

{ tools }

, adapt it to accept

{ ctx }

or call the real tool via the

tools

map provided by the runtime so the correct context is used. • Programmatic delegation (calling

realTool.execute

) is more reliable than returning instructions and hoping the LLM calls the real tool next. • If you cannot pass the same

ctx

, another safe option is for the wrapper to return a structured instruction that will cause the LLM to call the real tool next — but that relies on the LLM following the instruction. Alternative: use the runtime tools map (if supported) • Some runtimes give you a

tools

object in the execute args. You can call

await tools.real_tool.execute({ input }, { ctx })

to delegate while preserving the execution context. If you want, I can: • Convert the example to a TypeScript-typed version matching your project setup, or • Show a version that uses the

tools

map and demonstrates exact

ctx

passing depending on the SDK signature your runtime uses (e.g.,

{ session, ctx, tools }

). Which would help you more — TypeScript or a version showing

tools.real_tool.execute(...)

with explicit

ctx

passing? Sources: Tool definition and use | LiveKit Documentation