what is the best way to ensure the agent plays a message when executing a tool? should I put in the instructions or in the beginning of the tool?

Thanks for reaching out, Ahron. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

do tools have instructions or just a description?

Thanks — I’m responding to your two questions: (1) whether tools have "instructions" or just a description, and (2) where to put the agent’s spoken message so it reliably plays when a tool runs. Short answers • Tools have a description, not a separate "instructions" field like an agent prompt. The description tells the LLM the tool’s purpose and how to use its arguments. • To guarantee a message is spoken when a tool executes, generate the speech inside the tool’s execution code (use

session.generate_reply()

for contextual replies or

session.say()

for fixed playback). Why the description vs. instructions distinction matters • The tool description helps the LLM decide whether and how to call the tool. It does not cause the agent to speak when the tool runs. • Relying only on wording in the description to prompt the LLM to speak is unreliable—the LLM may still call the tool silently. How to implement (recommended) • Put the speak/play call at the start of the tool’s execute function so the message always runs when the tool runs. • Use: ◦

session.say()

for deterministic, short, fixed messages (e.g., “Connecting you now”). ◦

session.generate_reply()

for context-aware, natural responses generated by the LLM. Examples 1. Python (tool speaks then acts)

@function_tool()
async def transfer_to_billing(self, context: RunContext):
    await context.session.generate_reply(
        instructions="I'm transferring you to our billing department now."
    )
    # ...perform transfer...

1. TypeScript (tool speaks then acts)

const transferToBilling = llm.tool({
  description: 'Transfer to billing specialist.',
  execute: async (_, { ctx }) => {
    await ctx.session.generateReply({ instructions: "I'm transferring you to billing now." });
    // ...perform handoff...
  },
});

When to choose which approach • If you want guaranteed playback regardless of model behavior: put the message in the tool code. • If you want a dynamic, context-sensitive announcement: use

generate_reply()

inside the tool. • If you want a short, fixed announcement with no model generation: use

say()

inside the tool. If you’d like, I can: • Provide a full code sample tailored to your project (Python or TypeScript). • Suggest where to place messages for long-running tools, multi-step workflows, or handoffs. Which language and scenario should I show? Sources: Tool definition and use | LiveKit Documentation | Collect recording consent with tasks | LiveKit Documentation

this doesn't see to wait for playout

export function getSipHangupTool(config: HangupToolConfig) {
    return llm.tool({
        description: config?.tool_description || 'Hngup the phone call.',
        execute: async (_, { ctx }) => {
            const jobCtx = getJobContext();
            console.log('SIP Hangup Tool: Starting execution.');
            
            if (config.speak_during_execution && config?.speak_message) {
                await ctx.waitForPlayout();

                if (config.speak_type === 'prompt') {
                    ctx.session.generateReply({
                        instructions: config.speak_message,
                    })
                } else {
                    ctx.session.say(config.speak_message);
                }

                await ctx.waitForPlayout();
            }
            
            try {
                await hangupSipCall(jobCtx);
                return `SIP call has been hung up successfully.`;
            } catch (err) {
                const message =
                    err instanceof Error ? err.message : 'SIP transfer failed.';
                return `Failed to hang up SIP call: ${message}`;
            }
        },
    });
}

Thanks — I’m responding to your code showing that playout doesn’t seem to wait. You’re on the right track calling

waitForPlayout()

, but the issue is that

ctx.session.generateReply()

and

ctx.session.say()

are asynchronous and must be awaited (or otherwise ensured they’ve started) so playout actually begins before you call

waitForPlayout()

again. What’s happening in your code • You call

await ctx.waitForPlayout()

before initiating speech — that waits for any prior playout to finish (fine). • Then you call

ctx.session.generateReply({...})

or

ctx.session.say(...)

but do not await them. Those return promises that start the generation/playout; because you don’t await, the next line

await ctx.waitForPlayout()

may run before the speech has actually started (or before the generation is enqueued), so it returns immediately and the tool proceeds to hang up the call. What to change • Await the call that triggers playout so that the speech generation is scheduled before you wait for playout to complete. • For

generateReply()

you should await the returned promise (it resolves when the reply is generated/enqueued). • For

say()

you should also await it if it returns a promise (or otherwise ensure it is started). Suggested fix (TypeScript) 1. Await the generation/play call. 2. Then await

waitForPlayout()

to ensure the audio finishes before proceeding. Example adjusted snippet:

if (config.speak_during_execution && config?.speak_message) {
  await ctx.waitForPlayout(); // wait for prior playout to finish

  if (config.speak_type === 'prompt') {
    // await the generation so playout is enqueued
    await ctx.session.generateReply({
      instructions: config.speak_message,
    });
  } else {
    // await say() so playback is started/enqueued
    await ctx.session.say(config.speak_message);
  }

  // now wait until the newly enqueued playout finishes
  await ctx.waitForPlayout();
}

Additional tips • If

generateReply()

returns a reply object rather than a promise that only resolves when audio is enqueued in your SDK version, ensure you call whatever API signals the session to start playout. In that case, please share the return shape and we can adapt the awaits. • Add logging around each await to confirm ordering (e.g., log before/after generateReply and before/after waitForPlayout). • If speech generation can take non-trivial time and you want the hangup only after audio finishes, this pattern (await generate/say then await waitForPlayout) is the correct approach. If you try the above and it still doesn’t wait, tell me: • Which SDK/version you’re using, • Whether

generateReply()

or

say()

return a promise or an object, • Any logs you see for the before/after await points. I’ll help adapt the snippet to your exact SDK behavior.

using speech handle await handle.waitForPlayout(); gives an error

16:51:52.783] ERROR (2407689): exception occurred while executing tool
    function: "hangup"
    speech_id: "speech_80444ba0-f51"
    error: "Cannot call 'SpeechHandle.waitForPlayout()' from inside the function tool 'hangup'. This creates a circular wait: the speech handle is waiting for the function tool to complete, while the function tool is simultaneously waiting for the speech handle.\nTo wait for the assistant's spoken response prior to running this tool, use RunContext.wait_for_playout() instead."
[16:51:52.784] DEBUG (2407689): tools execution completed
    speech_id: "speech_80444ba0-f51"
[16:51:52.785] DEBUG (hangup/2407689): Tool call execution finished

however, await ctx.waitForPlayout(); didn't seem to work

Thanks — I’m responding to the error you hit and the fact

ctx.waitForPlayout()

didn't seem to work. What’s happening • You hit a circular-wait error because you tried to await a SpeechHandle’s

waitForPlayout()

from inside the same function tool that was initiated while the assistant was speaking. The SDK prevents that: the speech handle is waiting for the tool to finish, and the tool is waiting for the speech handle → circular wait. • The SDK message says to use

RunContext.wait_for_playout()

(in your code:

ctx.waitForPlayout()

) to wait for the assistant’s spoken response prior to running the tool. That is the correct API to use — but it must be called at the right time and around the right operations. Why your earlier

ctx.waitForPlayout()

“didn’t work” • In your first snippet you called

ctx.session.generateReply()

or

ctx.session.say()

without awaiting them. Because those calls weren’t awaited, the playout may not have been enqueued when you next called

await ctx.waitForPlayout()

, so it returned immediately. • Then you tried awaiting the SpeechHandle directly (e.g.,

handle.waitForPlayout()

), which caused the circular wait error when invoked from inside the same tool execution. Correct pattern to avoid circular waits 1. Ensure the session method that enqueues speech is awaited so the speech is actually scheduled (or returns a handle immediately that indicates scheduling has started). 2. Use

await ctx.waitForPlayout()

from the RunContext to wait for the enqueued playout to finish. Do not call

SpeechHandle.waitForPlayout()

from inside the tool — that causes the circular wait. Recommended TypeScript pattern • Await the call that enqueues speech, then await `ctx.waitForPlayout()`:

if (config.speak_during_execution && config?.speak_message) {
  // Wait for any prior playout to complete before starting new speech
  await ctx.waitForPlayout();

  // Enqueue the speech and await the enqueue operation so playout is actually scheduled
  if (config.speak_type === 'prompt') {
    await ctx.session.generateReply({ instructions: config.speak_message });
  } else {
    await ctx.session.say(config.speak_message);
  }

  // Wait until the newly-enqueued playout finishes
  await ctx.waitForPlayout();
}

Notes and troubleshooting • If

ctx.session.generateReply()

or

ctx.session.say()

do not return a promise that resolves when the speech is enqueued in your SDK version, you must call whatever API the SDK exposes that guarantees the speech was scheduled. If those methods already return such a promise, awaiting them is sufficient. • Add logging around these awaits to verify ordering: ◦ Log before/after `generateReply()`/`say()`. ◦ Log before/after

ctx.waitForPlayout()

. This will show whether playout was enqueued and when it completes. • Do not use the SpeechHandle’s

waitForPlayout()

inside the tool. That method is intended to be used from outside the tool execution (e.g., by the assistant runtime before invoking the tool), which is why it triggers the circular-wait protection here. If this still doesn’t wait as expected, please share: • The exact SDK and version you’re using. • Whether

generateReply()

and

say()

return promises (and what they resolve to). • The logs from the before/after points you added. I’ll review those and help adapt the pattern to your SDK behavior.

await ctx.waitForPlayout(); does not wait. If I add afterwards a small timeout I can hear a bit of the message

export function getSipHangupTool(config: HangupToolConfig) {
    return llm.tool({
        description: config?.tool_description || 'Hngup the phone call.',
        execute: async (_, { ctx }) => {
            const jobCtx = getJobContext();
            console.log('SIP Hangup Tool: Starting execution.');
            
            if (config.speak_during_execution && config?.speak_message) {
                await ctx.waitForPlayout();

                if (config.speak_type === 'prompt') {
                    ctx.session.generateReply({
                        instructions: config.speak_message,
                    });
                } else {
                    ctx.session.say(config.speak_message);
                }

                await ctx.waitForPlayout(); //does not wait here
                await new Promise((resolve) => setTimeout(resolve, 1000));
            
            }
            
            try {
                await hangupSipCall(jobCtx);
                return `SIP call has been hung up successfully.`;
            } catch (err) {
                const message =
                    err instanceof Error ? err.message : 'SIP transfer failed.';
                return `Failed to hang up SIP call: ${message}`;
            }
        },
    });
}

Thanks — I’m replying about your latest snippet where

ctx.waitForPlayout()

does not wait (you only hear audio if you add a timeout afterwards). Short answer: the issue is that the calls that enqueue speech (like

ctx.session.generateReply()

or

ctx.session.say()

) are not being awaited, so playout may not be scheduled by the time you call

await ctx.waitForPlayout()

. You must await the enqueue call so the session actually starts the speech before waiting for it to finish. Also, make sure the SDK methods you call do indeed return a promise that resolves when the speech is enqueued — if they don’t, you need to use whatever enqueue-confirmation API your SDK exposes. What’s likely happening in your code • You call

await ctx.waitForPlayout()

to clear prior audio (good). • You then call

ctx.session.generateReply({...})

or

ctx.session.say(...)

without awaiting them. Those calls start async work to generate/enqueue audio. • Immediately after,

await ctx.waitForPlayout()

runs but the speech hasn’t been enqueued yet, so it returns immediately (no playout to wait for). • Adding a manual timeout gives the SDK time to start enqueuing and playing, which is why you hear part of the message. Concrete changes to make 1. Await the enqueue call (both for prompt-generated replies and

say()

). 2. Then await

ctx.waitForPlayout()

to wait until playback completes. Updated snippet (TypeScript):

if (config.speak_during_execution && config?.speak_message) {
  // Wait for any prior playout
  await ctx.waitForPlayout();

  // Enqueue speech and await the enqueue operation
  if (config.speak_type === 'prompt') {
    await ctx.session.generateReply({
      instructions: config.speak_message,
    });
  } else {
    await ctx.session.say(config.speak_message);
  }

  // Wait until the newly-enqueued playout finishes
  await ctx.waitForPlayout();
}

If that still doesn’t wait, next checks to run • Confirm the return type/behavior: ◦ Does

ctx.session.generateReply()

return a promise that resolves once the reply is enqueued (not only when text is generated)? Does

ctx.session.say()

return a promise? If either method does not return an enqueue-complete promise, `await`ing them won’t guarantee playout is scheduled. • Add logging to verify ordering: ◦ Log before/after the enqueue await and before/after

ctx.waitForPlayout()

. ◦ This will tell you whether enqueueing is happening and when playout finishes. • Ensure there’s no separate "start playout" step in your SDK version. Some SDKs return a handle you need to start or confirm. If you confirm that `generateReply()`/`say()` are not resolving when audio is enqueued, share: • The SDK name and exact version you’re using. • What those methods return (a promise, an object, a handle — copy/paste the returned object shape if possible). With that I can show the exact pattern to guarantee playout completes before you hang up the SIP call.

no

waitForPlayout

is not a promise. please search and find the solution for me

Thanks — I’m replying to your latest note: you said

waitForPlayout

is not a promise and that

ctx.waitForPlayout()

still does not wait in your tool. I hear your frustration — let’s get this working reliably. Summary of the correct, reliable approach • Do not call a SpeechHandle’s

waitForPlayout()

from inside the function tool if that SpeechHandle was created by the same assistant turn — that causes the circular-wait error. • The reliable way is to capture the SpeechHandle returned by

session.generateReply()

or

session.say()

and then await the SpeechHandle (or its

waitForPlayout()

method). This ensures the speech is scheduled and you can wait for playback to finish before continuing. • If

ctx.waitForPlayout()

is not pausing for you, it usually means speech was not properly enqueued before you called it. The fix is to await the SpeechHandle (or ensure the enqueue step has completed) before calling

ctx.waitForPlayout()

. Concrete Node.js pattern to use in your tool 1. Enqueue speech and capture the returned handle. 2. Await the handle (or await handle.waitForPlayout()) so playback is actually scheduled and you wait for its completion. 3. Then proceed to hang up. Example adjusted code for your hangup tool (TypeScript):

export function getSipHangupTool(config: HangupToolConfig) {
  return llm.tool({
    description: config?.tool_description || 'Hang up the phone call.',
    execute: async (_, { ctx }) => {
      const jobCtx = getJobContext();
      console.log('SIP Hangup Tool: Starting execution.');

      if (config.speak_during_execution && config?.speak_message) {
        // Wait for any prior playout to finish
        await ctx.waitForPlayout();

        // Enqueue the speech and capture the SpeechHandle
        let handle;
        if (config.speak_type === 'prompt') {
          handle = await ctx.session.generateReply({
            instructions: config.speak_message,
          });
        } else {
          handle = await ctx.session.say(config.speak_message);
        }

        // Await the handle so we wait until playback is complete
        // either by awaiting the handle directly or calling its waitForPlayout()
        if (handle && typeof handle.waitForPlayout === 'function') {
          await handle.waitForPlayout();
        } else {
          // SpeechHandle is awaitable in some SDKs
          await handle;
        }
      }

      try {
        await hangupSipCall(jobCtx);
        return `SIP call has been hung up successfully.`;
      } catch (err) {
        const message =
          err instanceof Error ? err.message : 'SIP transfer failed.';
        return `Failed to hang up SIP call: ${message}`;
      }
    },
  });
}

Why this fixes the behavior • Awaiting `generateReply()`/`say()` (or the returned SpeechHandle) ensures the SDK has actually enqueued the TTS/playout. • Awaiting the SpeechHandle (or calling its

waitForPlayout()

) ensures your code pauses until audio finishes. • Using

ctx.waitForPlayout()

is still useful to wait for already-enqueued speech, but it must be used after the enqueue step — not instead of awaiting the SpeechHandle when enqueueing happens in the same turn. If that still doesn’t work, please check and share: • The exact SDK package and version you’re using (e.g., livekit-agents vX.Y.Z). • What

ctx.session.generateReply()

and

ctx.session.say()

return in your runtime (copy/paste the returned object or its shape). • Logs around the steps (add console logs before/after enqueue and before/after await) so we can see ordering. I’ll use that to produce the minimal change needed for your SDK version. Do you want me to tailor the snippet for a specific LiveKit Agents version? Sources: Agent speech and audio | LiveKit Documentation | How to detect when an agent has finished speaking | LiveKit