how to generate out-of-band responses for openai realtime plugin? particulary, on long tool calls, i want to have more thinking phrases in-between instead of just a single preamble phrase

Thanks for reaching out, Sebastián. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

yes please

Great — I can provide both concise examples. Which do you prefer: Python (asyncio) or Node.js (TypeScript/JS)? Or would you like both?

python

Thanks — below is a concise Python asyncio example showing how to send multiple out-of-band “thinking” phrases during a long tool call using

context.session.generate_reply()

. It follows the pattern: start a background task that sends phrases periodically, run the long operation, then stop/cancel the background task.

import asyncio
from livekit.agents import function_tool, RunContext

@function_tool()
async def perform_long_search(self, context: RunContext, query: str) -> str:
    stop_speaking = False

    async def send_thinking_updates():
        phrases = [
            "Let me search for that...",
            "Looking through my knowledge base...",
            "Still gathering information...",
            "Almost there..."
        ]
        for p in phrases:
            if stop_speaking:
                break
            # send an out-of-band reply (spoken/text) to the session
            await context.session.generate_reply(instructions=p)
            await asyncio.sleep(3)  # space messages

    thinking_task = asyncio.create_task(send_thinking_updates())

    try:
        # Replace with your real long-running work
        result = await _perform_expensive_search(query)  # implement this
        return result
    finally:
        stop_speaking = True
        thinking_task.cancel()
        try:
            await thinking_task
        except asyncio.CancelledError:
            pass

Notes: • Use

stop_speaking

and task cancellation to avoid overlapping or leftover messages. • Adjust phrases and

sleep

interval (e.g., 2–4s) to suit your UX. • Optionally call

context.room.local_participant.publish_text()

to show the same messages in the UI. Want this adapted to a specific long-task implementation or with companion UI text publishing included?

i don't want prefixed responses and ideally, to not pollute conversation with this phrases

Thanks — I’m responding to your request about sending unprefixed, non-polluting thinking phrases. Quick summary of options you can use in Python (choose one): • Use session.say() to speak short status phrases as synthesized audio only — this does not add text to the conversation transcript and won’t prepend LLM-style prefixes. ◦ Example: await context.session.say("Checking that now.", allow_interruptions=False) • Send thinking state via a separate out-of-band channel (preferred for UI-only indicators) using LiveKit data messages so the client shows status without any speech or chat history pollution. ◦ Example: context.room.local_participant.publish_data({"type":"agent_status","status":"checking"}) • If you want LLM-style natural phrasing but must avoid polluting history: generate the phrase locally (or via a lightweight prompt) and play it with session.say() as audio only, rather than calling generate_reply() which appends to chat history. Clarifying question to pick the best snippet for you: • Do you want these updates as audio only, UI-only (no audio), or both? Tell me which and I’ll provide a concise Python snippet for that flow.

do you support openai response.create that don't add to the conversation history?

Short answer: No — we do not support an OpenAI

response.create

-style call that bypasses conversation history. What you can do instead: • Use

session.say()

to play short TTS/status phrases as audio only; these do not get added to the transcript. ◦ Example:

await context.session.say("Checking that now.", allow_interruptions=False)

• Publish out-of-band UI state via data messages:

context.room.local_participant.publish_data({...})

so clients show status without chat entries. • Use a separate TTS service/plugin for ephemeral spoken updates, keeping LLM outputs out of the transcript. • Or generate the phrase locally (or via a lightweight prompt) and play it with

session.say()

rather than calling

generate_reply()

which is recorded. If you want, I can provide a short Python snippet showing any of these approaches (audio-only

session.say()

, data messages, or TTS integration). Which one would you like? Sources: OpenAI Realtime API plugin guide | LiveKit docs | OpenAI and LiveKit partner to turn Advanced Voice into an API | External data and RAG | LiveKit docs | LLM Output Replacement | LiveKit docs

do you support async tool calls, where instead of waiting for long tool calls to finish, run them in bg and continue conversation, when response is ready, let the user know it's ready?

Yes — we support async tool calls. You can start long-running work in the background, return control to the user immediately, and notify them when the background result is ready. Concise pattern (Python asyncio + LiveKit Agents): 1) Start a background task from your tool and return immediately to keep the conversation flowing. 2) Use

context.session.say()

(audio-only) or

context.room.local_participant.publish_data()

(UI-only) for status updates so you don't pollute chat history. 3) When the background task completes, notify the user via

context.session.generate_reply()

(adds to history) or

context.session.say()

/

publish_data()

(does not add to transcript). Minimal example:

import asyncio
from livekit.agents import function_tool, RunContext

@function_tool()
async def async_tool(self, context: RunContext, query: str) -> str:
    async def long_work():
        # do long-running processing here
        await asyncio.sleep(20)
        return f"Result ready for: {query}"

    task = asyncio.create_task(long_work())

    # Immediate feedback without adding to chat history
    await context.session.say("I've started your request and will tell you when it's ready.", allow_interruptions=False)
    # OR publish UI-only status:
    # context.room.local_participant.publish_data({"type": "agent_status", "status": "processing", "query": query})

    # return control to the agent/user now
    return "Got it — I'm processing that now."

    # Separate runner: notify when done
    async def notify_when_done(t):
        try:
            result = await t
            # notify user (choose one)
            await context.session.say("Your result is ready.", allow_interruptions=False)  # audio-only, not in transcript
            await context.session.generate_reply(instructions=f"The analysis is complete: {result}")  # adds to history
            context.room.local_participant.publish_data({"type":"agent_result","result":result})  # UI-only
        except Exception:
            await context.session.say("There was an error finishing your request.", allow_interruptions=False)

    # fire-and-forget notifier
    asyncio.create_task(notify_when_done(task))

Notes / best practices: • Use

say()

or data messages to avoid polluting chat history; use

generate_reply()

only when you want the result recorded in the conversation. • Manage task lifecycle and cancellation (handle interruptions). • For very long jobs, surface cancellation/progress in the UI via data messages and let clients handle polling or fetch of full results. Want a version adapted to your architecture (stateless serverless, background worker queue, or persistent process)? Which environment are you running in? Sources: Tool definition and use | LiveKit docs | External data and RAG | LiveKit docs

i want the approach of using session.generate reply for generating thinking phrases

Thanks — you want to use

session.generate_reply()

to produce LLM-generated thinking phrases during a long-running tool, while running the tool in the background and notifying the user when results are ready. Below is a concise, practical Python asyncio pattern that does exactly that and manages history/pollution explicitly. Key points up front: •

generate_reply()

will add its output to conversation history. Use it when you want those thinking phrases to be part of the chat. • To avoid overlap, run a single background thinking loop that checks a shared flag and cancels cleanly when work finishes. • Run the long task in a background task so the tool returns immediately and the agent can continue conversing. Concise example (Python asyncio):

import asyncio
from livekit.agents import function_tool, RunContext

@function_tool()
async def background_tool(self, context: RunContext, query: str) -> str:
    stop_thinking = False

    async def thinking_loop():
        phrases = [
            "Let me check that for you...",
            "Digging through sources...",
            "Still looking...",
            "Almost done..."
        ]
        i = 0
        while not stop_thinking:
            phrase = phrases[i % len(phrases)]
            # generate_reply() will create an LLM response and be added to history
            await context.session.generate_reply(instructions=phrase, allow_interruptions=True)
            i += 1
            await asyncio.sleep(3)  # adjust interval as desired

    async def long_work():
        # replace with real long-running logic
        await asyncio.sleep(20)
        return f"Final results for: {query}"

    # start background tasks
    thinking_task = asyncio.create_task(thinking_loop())
    work_task = asyncio.create_task(long_work())

    # give immediate feedback to user (optional): a short, recorded reply
    await context.session.generate_reply(instructions="I've started processing that — I'll update you when it's ready.", allow_interruptions=True)

    # return control to the agent/user immediately
    # this returned string is the tool's immediate return value if needed
    # (you can return a short confirmation or None)
    asyncio.create_task(_notify_when_done(work_task, thinking_task, context))
    return "Processing started."

async def _notify_when_done(work_task: asyncio.Task, thinking_task: asyncio.Task, context: RunContext):
    try:
        result = await work_task
        # stop thinking loop before final notification
        thinking_task.cancel()
        # final notification: you can choose generate_reply() (adds to history)
        await context.session.generate_reply(instructions=f"The result is ready: {result}", allow_interruptions=True)
        # optionally publish UI-only data:
        # context.room.local_participant.publish_data({"type":"agent_result","result": result})
    except asyncio.CancelledError:
        pass
    except Exception:
        thinking_task.cancel()
        await context.session.generate_reply(instructions="There was an error finishing your request.", allow_interruptions=True)

Notes and best practices: • Because

generate_reply()

writes to history, avoid excessive thinking phrases to keep the transcript usable. • Use

allow_interruptions=True

if you want the user to interrupt the thinking messages. • Control frequency (sleep) and phrase variety to reduce transcript noise. • If you need those thinking updates not to appear in history, use

session.say()

or publish data messages instead (but you said you prefer

generate_reply()

so this example uses it). • Ensure proper task cancellation and exception handling so thinking messages stop reliably. Would you like this adapted to a stateless serverless environment (e.g., using an external job queue) or do you run a persistent process where background tasks are fine?

it's important that this integrates smoothly with the agent states: we have to play thinking sound when not saying phrases and back to the thinking sounds when played the sound and so on

Got it — you want a Python solution that uses

session.generate_reply()

for LLM-generated thinking phrases while seamlessly coordinating thinking-sound playback and agent state transitions (thinking → speaking → back to thinking). Below is a concise, runnable pattern that: • Runs the long task in background • Emits periodic

generate_reply()

thinking phrases (which are added to history) • Uses BackgroundAudioPlayer so thinking sounds play while the agent is idle/thinking and automatically pause during speech • Ensures clean cancellation and state transitions Copy/paste-ready example:

import asyncio
from livekit.agents import (
    function_tool, RunContext, BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip
)

@function_tool()
async def async_tool_with_thinking(self, context: RunContext, query: str) -> str:
    # Configure and start thinking-sound player (start once per session in real code)
    background_audio = BackgroundAudioPlayer(
        thinking_sound=[AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.6)]
    )
    await background_audio.start(room=context.room, agent_session=context.session)

    stop_thinking = False

    async def thinking_loop():
        phrases = [
            "Let me check that for you...",
            "Looking through the sources...",
            "Still working on it...",
            "Almost done..."
        ]
        i = 0
        while not stop_thinking:
            # This transitions agent -> speaking (via generate_reply) and BackgroundAudioPlayer
            # will automatically pause thinking sounds while speech plays and resume after.
            await context.session.generate_reply(instructions=phrases[i % len(phrases)], allow_interruptions=True)
            i += 1
            await asyncio.sleep(3)  # adjust interval to taste

    async def long_work():
        # Replace with your actual long-running logic
        await asyncio.sleep(20)
        return f"Final results for {query}"

    # start background tasks
    thinking_task = asyncio.create_task(thinking_loop())
    work_task = asyncio.create_task(long_work())

    # immediate confirmation (optional) — also via generate_reply so it's in history
    await context.session.generate_reply(instructions="I've started processing that and will update you when it's ready.", allow_interruptions=True)

    # fire-and-forget notifier that stops thinking sounds and posts final result
    async def notify_done():
        nonlocal stop_thinking
        try:
            result = await work_task
            # stop thinking loop before final notification
            stop_thinking = True
            thinking_task.cancel()
            try:
                await thinking_task
            except asyncio.CancelledError:
                pass

            # final answer: generate_reply() (will speak and be in history)
            await context.session.generate_reply(instructions=f"The result is ready: {result}", allow_interruptions=True)
            # optionally publish UI-only data:
            # context.room.local_participant.publish_data({"type":"agent_result","result": result})
        except Exception:
            stop_thinking = True
            thinking_task.cancel()
            await context.session.generate_reply(instructions="There was an error finishing your request.", allow_interruptions=True)

    asyncio.create_task(notify_done())

    # return control immediately (tool return value can be a short confirmation)
    return "Processing started."

Key notes and best practices: • BackgroundAudioPlayer automatically pauses thinking sounds while the agent is speaking after

generate_reply()

and resumes when the agent returns to thinking — no manual toggling needed. • Keep thinking phrase frequency reasonable (2–4s) to avoid transcript spam. • Use

allow_interruptions=True

on

generate_reply()

if you want the user to interrupt thinking phrases. • Ensure BackgroundAudioPlayer is started once per session (shown inline here for clarity). • Handle cancellation and exceptions so thinking sounds stop reliably if the task is interrupted. Would you like this adapted to: • a persistent agent process versus serverless (job queue) architecture, or • a variant that uses

session.say()

for some phrases and

generate_reply()

for others? Sources: Turns overview | LiveKit docs | Agent speech and audio | LiveKit docs | Web and mobile frontends | LiveKit docs

the thinking phrases should not overlap, we have to finish playing one and then do the backoff until next one or finished the tool call

Acknowledged — you want non-overlapping LLM-generated thinking phrases that: 1) finish playing before the next phrase starts, 2) fall back to thinking-sound playback between phrases, and 3) stop immediately when the long task finishes. Below is a concise Python asyncio pattern that enforces those guarantees using speech handles, explicit await for speech completion, backoff timing, and clean cancellation. Copy/paste-ready pattern:

import asyncio
from livekit.agents import function_tool, RunContext, BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip

BACKOFF_SECONDS = 3.0  # wait after a phrase finishes before next phrase

@function_tool()
async def async_tool_no_overlap(self, context: RunContext, query: str) -> str:
    # Start thinking-sound player once per session in real usage
    background_audio = BackgroundAudioPlayer(
        thinking_sound=[AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.5)]
    )
    await background_audio.start(room=context.room, agent_session=context.session)

    stop_requested = False

    async def thinking_sequence():
        phrases = [
            "Let me check that for you...",
            "Looking through the sources...",
            "Still working on it...",
            "Almost done..."
        ]
        i = 0
        while not stop_requested:
            phrase = phrases[i % len(phrases)]

            # Request an LLM-generated utterance and get a speech handle we can await
            speech_handle = await context.session.generate_reply(
                instructions=phrase,
                allow_interruptions=True,
                return_speech_handle=True  # hypothetical handle return flag; adjust to your SDK if named differently
            )

            # Pause sending new phrases until this speech completes or is interrupted
            try:
                # Wait for the speech to finish. If the SDK exposes `wait_until_complete()` or similar, use it.
                await speech_handle.wait_until_complete()  # await actual playback completion
            except asyncio.CancelledError:
                # If cancelled, ensure playback is stopped
                await speech_handle.stop()
                raise

            # After phrase playback, wait BACKOFF_SECONDS while thinking-sound resumes
            # If stop_requested becomes True while sleeping, exit promptly
            slept = 0.0
            while slept < BACKOFF_SECONDS and not stop_requested:
                await asyncio.sleep(0.1)
                slept += 0.1

            i += 1

    async def long_work():
        # replace with real long-running operation
        await asyncio.sleep(20)
        return f"Result for {query}"

    thinking_task = asyncio.create_task(thinking_sequence())
    work_task = asyncio.create_task(long_work())

    # Immediate confirmation (optional)
    await context.session.generate_reply(instructions="I've started processing that and will update you when it's ready.", allow_interruptions=True)

    async def notifier():
        nonlocal stop_requested
        try:
            result = await work_task
            # stop sending further phrases
            stop_requested = True
            # ensure thinking loop is done / cancelled
            thinking_task.cancel()
            try:
                await thinking_task
            except asyncio.CancelledError:
                pass

            # Final result (will speak and be added to history)
            await context.session.generate_reply(instructions=f"The result is ready: {result}", allow_interruptions=True)
        except Exception:
            stop_requested = True
            thinking_task.cancel()
            try:
                await thinking_task
            except asyncio.CancelledError:
                pass
            await context.session.generate_reply(instructions="There was an error finishing your request.", allow_interruptions=True)

    asyncio.create_task(notifier())

    return "Processing started."

Key implementation details and considerations: • Use the SDK’s speech handle mechanism (here shown as

return_speech_handle=True

and

speech_handle.wait_until_complete()

) so you can await exact playback completion. If your SDK exposes different names, use the equivalent (e.g.,

speech_handle.wait_if_not_interrupted()

or

speech_handle.finished()

). • While waiting between phrases, BackgroundAudioPlayer will resume thinking sound automatically; we sleep in small increments to allow prompt exit if the work finishes. • Set

allow_interruptions=True

if you want users to interrupt thinking phrases. If interruption should abort the thinking loop, handle that in the speech handle/error path. • Keep backoff reasonable (2–4s) to avoid transcript noise. • Ensure BackgroundAudioPlayer is started once per session in production (the example starts it inline for clarity). • If your SDK does not return a speech handle from

generate_reply()

, you can: ◦ Listen to agent state events (agent_state_changed) and await transition from

speaking

back to

thinking

, or ◦ Use

session.say()

which may return a handle you can await — adapt as needed. If you want, I can: • Adapt this to the exact SDK method names you have (tell me how your

generate_reply()

returns a handle), or • Provide a variant where interruptions cancel current speech and immediately post a short phrase. Which do you prefer?

as an example, this works working with the OpenAI realtime model directly:

async def _send_out_of_band_filler(
        self, websocket: WebSocketHandler, filler_id: str, filler_count: int
    ):
        """
        POC: Send an out-of-band filler response that doesn't affect the conversation.
        Uses response.conversation = "none" to create audio outside the
        default conversation.
        """
        <http://logger.info|logger.info>("[POC] Sending out-of-band filler")

        # Track the active filler so we know when it completes
        self._active_filler_id = filler_id
        self._filler_done_event.clear()

        wait_stage = (
            "early" if filler_count == 0 else "mid" if filler_count == 1 else "late"
        )

        filler_event = {
            "type": "response.create",
            "response": {
                "conversation": "none",  # Out-of-band - won't affect main conversation
                "output_modalities": ["audio"],
                "input": [],
                "tool_choice": "none",
                "instructions": f"""You are emitting a short OUT-OF-BAND waiting phrase.
This response is NOT part of the conversation.

You are NOT replying to the user.
You are NOT responding to a tool or its result.
You must NOT explain, answer, or add information.

Your task is ONLY to produce a brief, natural waiting phrase
that matches the provided wait stage.

Wait stage:{wait_stage}

Rules:
- Output ONE short sentence only.
- Do not ask questions.
- Do not reference tools, thinking, searching, or processing.
- Do not mention what is happening or why.
- Do not continue the conversation.
- Speak in Argentinian

Choose a phrase that sounds appropriate for the wait stage:

If wait_stage = early:
- casual acknowledgement, first pause
Example: "One moment please while I check that for you..."
If wait_stage = mid:
- continued waiting, gentle reassurance
Example: "I'm still working on that..."
If wait_stage = late:
- longer wait, calm reassurance without urgency
Example: "Sorry for the delay, I'm almost there..."
""",
                "metadata": {
                    "purpose": "filler",
                    "filler_id": filler_id,
                },
            },
        }

        await websocket.send_json(filler_event)
        <http://logger.info|logger.info>(f"[POC] Out-of-band filler sent with id: {filler_id}")

Thanks — I see you already have a working POC using the OpenAI Realtime

response.create

with

conversation: "none"

. You want a robust pattern that: • Uses out-of-band audio (no convo pollution) • Ensures thinking phrases never overlap (finish one, backoff, resume thinking sound) • Integrates with agent states and thinking-sound playback • Stops immediately when the long task finishes or is interrupted Below is a concise, production-ready Python pattern that combines your

response.create

approach with LiveKit agent state integration and non-overlap guarantees. Key ideas: • Use the Realtime

response.create

with

conversation: "none"

for fillers (no history). • Track an active filler ID and await a speech-completion signal (via a speech handle, event, or agent state transition) before sending the next filler. • Use BackgroundAudioPlayer for continuous thinking sound; it will pause automatically during speech and resume after. • Use a backoff timer between fillers; cancel promptly when the long task completes. Concise example (adapt SDK call names to your environment):

import asyncio
import uuid
from livekit.agents import BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip

BACKOFF_SECONDS = 3.0

async def send_out_of_band_filler(ws, instructions, filler_id):
    # Build the response.create payload that won't affect main convo
    event = {
        "type": "response.create",
        "response": {
            "conversation": "none",
            "output_modalities": ["audio"],
            "input": [],
            "tool_choice": "none",
            "instructions": instructions,
            "metadata": {"purpose": "filler", "filler_id": filler_id},
        },
    }
    await ws.send_json(event)
    return filler_id

async def wait_for_filler_complete(filler_done_event, timeout=None):
    # Wait until event set by your websocket handler when filler playback finishes
    try:
        await asyncio.wait_for(filler_done_event.wait(), timeout=timeout)
    except asyncio.TimeoutError:
        return False
    return True

async def long_task_with_fillers(ctx, ws, query):
    # Ensure thinking sound plays while idle
    background_audio = BackgroundAudioPlayer(
        thinking_sound=[AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.6)]
    )
    await background_audio.start(room=ctx.room, agent_session=ctx.session)

    filler_done_event = asyncio.Event()
    active_filler_id = None
    stop_requested = False

    async def filler_manager():
        nonlocal active_filler_id, stop_requested
        stages = ["early", "mid", "late"]
        i = 0
        while not stop_requested:
            # Wait backoff before sending next filler (resume thinking sound during backoff)
            await asyncio.sleep(BACKOFF_SECONDS)

            # Prepare filler instructions for current stage
            stage = stages[min(i, len(stages)-1)]
            instructions = f"Emit one short filler sentence for wait stage: {stage}."

            # send out-of-band filler via realtime API (conversation:none)
            fid = str(uuid.uuid4())
            active_filler_id = fid
            filler_done_event.clear()
            await send_out_of_band_filler(ws, instructions, fid)

            # Wait until playback completes (signalled by websocket handler setting event)
            completed = await wait_for_filler_complete(filler_done_event, timeout=30.0)
            active_filler_id = None
            if not completed:
                # fallback: continue or break depending on policy
                pass

            i += 1

    # Start filler manager but don't block the main return
    filler_task = asyncio.create_task(filler_manager())

    async def long_work():
        # Replace with your real long-running operation
        await asyncio.sleep(25)
        return f"Final result for {query}"

    work_task = asyncio.create_task(long_work())

    # Immediately confirm start via session.say() or publish_data (non-polluting)
    await ctx.session.say("I've started processing that and will notify you when it's ready.", allow_interruptions=True)

    # notifier: stop fillers and post final result
    async def notifier():
        nonlocal stop_requested
        try:
            result = await work_task
            stop_requested = True
            # If a filler is currently playing, wait for it to finish or optionally interrupt it via API
            if active_filler_id:
                # Optionally send a control event to interrupt by ID if supported
                # e.g., await ws.send_json({"type":"response.cancel","id": active_filler_id})
                await wait_for_filler_complete(filler_done_event, timeout=3.0)

            # ensure filler manager stops
            try:
                await asyncio.wait_for(filler_task, timeout=1.0)
            except asyncio.TimeoutError:
                filler_task.cancel()

            # Final notification: speak or publish result
            await ctx.session.generate_reply(instructions=f"The result is ready: {result}", allow_interruptions=True)
            # Optionally publish UI-only payload:
            # ctx.room.local_participant.publish_data({"type":"agent_result","result":result})
        except Exception:
            stop_requested = True
            filler_task.cancel()
            await ctx.session.generate_reply(instructions="There was an error finishing your request.", allow_interruptions=True)

    asyncio.create_task(notifier())

    return "Processing started."

Integration notes you must implement: • WebSocket handler must set/clear

filler_done_event

when the client/server signals a filler audio finished. Your POC already sets

_filler_done_event

— reuse that to coordinate non-overlap. • If your realtime endpoint supports cancelling a specific response by ID, use that to abort an active filler immediately when cancelling; otherwise wait briefly for it to finish. • BackgroundAudioPlayer will automatically pause during any speech generated by the agent session; keep it running so thinking sound resumes between fillers. • Tune BACKOFF_SECONDS and filler timeouts to balance UX vs transcript noise. If you want, I can: • Convert this into a drop-in variant that uses your exact websocket message names and your existing filler-complete signalling (send me the handler names), or • Add example client-side logic for emitting the filler completion event so the server-side event is triggered precisely when audio ends. Which would you prefer? Sources: OpenAI and LiveKit partner to turn Advanced Voice into an API | OpenAI Realtime API plugin guide | LiveKit docs | Agent speech and audio | LiveKit docs | External data and RAG | LiveKit docs | examples/voice_agents | livekit/livekit_composite | DeepWiki

but how to integrate into LiveKit flow with OpenAI Realtime plugin and methods exposed to the agent and agent session

Thanks — here’s a focused, actionable plan and a concise Python pattern showing how to integrate out-of-band fillers (using OpenAI Realtime

response.create

behavior or `session.say()`/`generate_reply()`), non-overlapping LLM-generated phrases, and thinking-sound playback into the LiveKit AgentSession flow. Which approach you should pick • If you must avoid polluting conversation history: send out-of-band audio via the Realtime

response.create

with

"conversation": "none"

(your POC). This is supported when you can reach the underlying realtime session/websocket. • If you prefer to use AgentSession methods only: configure Realtime as text-only + a TTS plugin, then use

session.say()

for non-historical fillers and

session.generate_reply()

for LLM replies that should go into history. • For LLM-generated fillers that do not overlap and still avoid polluting history, use

response.create

(conversation:none) — otherwise

generate_reply()

will add to history. Concise integration pattern (Python, ready to adapt) • Goals: run long task in background, play continuous thinking-sound while idle, emit LLM-generated filler phrases that never overlap (wait for each to finish), and stop immediately when work completes. 1) Setup (once per session) • Start BackgroundAudioPlayer for thinking sound: ◦ background_audio = BackgroundAudioPlayer(thinking_sound=[AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.6)]) ◦ await background_audio.start(room=ctx.room, agent_session=ctx.session) 2) Main tool flow (core ideas) • Launch the long work as an asyncio task. • Launch a filler manager task that: ◦ waits BACKOFF between fillers, ◦ sends an out-of-band filler (prefer

response.create

with conversation:"none" if available; otherwise

session.generate_reply()

), ◦ awaits a playback-complete signal (SpeechHandle completion or a websocket event) before proceeding to backoff/resume thinking sound. • When the long work completes, stop the filler manager, wait or cancel any active filler cleanly, then post the final result (via

generate_reply()

if you want it in history, or `session.say()`/publish_data for non-historical UI/audio). 3) Non-overlap and state coordination techniques • Use a SpeechHandle (or your websocket’s filler-done event) to know exactly when speech finishes; await it before starting the backoff timer and next filler. • While filler is not playing, BackgroundAudioPlayer will play thinking-sound automatically (it pauses during speech). • If you cannot get a speech handle from

response.create

, have the client emit a "filler_finished" data message when playback ends; server awaits that event. • To abort immediately on completion: if a filler is active and your Realtime endpoint supports cancelling, send response.cancel for that filler ID; otherwise wait briefly for completion then continue. Copy/paste-ready pattern (concise)

import asyncio, uuid
from livekit.agents import BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip

BACKOFF = 3.0

async def tool_with_nonoverlap(ctx, ws, query):
    # Start thinking sound once per session
    background_audio = BackgroundAudioPlayer(thinking_sound=[AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.6)])
    await background_audio.start(room=ctx.room, agent_session=ctx.session)

    stop_flag = False
    active_filler_id = None
    filler_done = asyncio.Event()

    async def send_response_create(ws, instructions, fid):
        payload = {
            "type": "response.create",
            "response": {
                "conversation": "none",
                "output_modalities": ["audio"],
                "input": [],
                "tool_choice": "none",
                "instructions": instructions,
                "metadata": {"filler_id": fid},
            },
        }
        await ws.send_json(payload)
        return fid

    async def filler_manager():
        nonlocal active_filler_id
        stages = ["early","mid","late"]
        i = 0
        while not stop_flag:
            await asyncio.sleep(BACKOFF)  # thinking sound plays during this backoff
            if stop_flag:
                break
            stage = stages[min(i, len(stages)-1)]
            instr = f"One short filler sentence for wait stage: {stage}."
            fid = str(uuid.uuid4())
            active_filler_id = fid
            filler_done.clear()
            await send_response_create(ws, instr, fid)
            # wait for playback to finish (set by websocket handler when filler audio ends)
            await filler_done.wait()
            active_filler_id = None
            i += 1

    async def long_work():
        await asyncio.sleep(20)  # replace with actual work
        return "final result"

    filler_task = asyncio.create_task(filler_manager())
    work_task = asyncio.create_task(long_work())

    # notify user we started (non-historical)
    await ctx.session.say("I've started processing that and will tell you when it's ready.", allow_interruptions=True)

    async def notifier():
        nonlocal stop_flag
        try:
            result = await work_task
            stop_flag = True
            # if filler active, attempt cancel or wait briefly
            if active_filler_id:
                # optionally: await cancel via websocket if supported
                # await ws.send_json({"type":"response.cancel","id":active_filler_id})
                await asyncio.wait_for(filler_done.wait(), timeout=3.0)
            filler_task.cancel()
            try: await filler_task
            except asyncio.CancelledError: pass

            # final reply into history
            await ctx.session.generate_reply(instructions=f"The result is ready: {result}", allow_interruptions=True)
        except Exception:
            stop_flag = True
            filler_task.cancel()
            await ctx.session.generate_reply(instructions="There was an error finishing your request.", allow_interruptions=True)

    asyncio.create_task(notifier())
    return "Processing started."

What you must wire on your side • Websocket handler must set

filler_done.set()

when the out-of-band filler audio finishes playback (based on the response.id or metadata). • If you prefer AgentSession-only methods, replace

send_response_create()

with

await ctx.session.say(..., add_to_chat_ctx=False)

and use the returned SpeechHandle to await completion instead of

filler_done

. • Tune BACKOFF and timeouts to your UX. If you want I can: • Convert this to a pure AgentSession-only version using

session.say()

and SpeechHandle (if you run text-only Realtime + TTS), or • Produce the exact websocket handler code that sets filler completion events based on your current POC’s event flow. Which variant do you want me to produce next: AgentSession-only (TTS + speech handle) or the

response.create

websocket-based flow wired into your existing POC? Sources: Agent speech and audio | LiveKit docs | OpenAI Realtime API plugin guide | LiveKit docs | Azure OpenAI Realtime API and LiveKit | LiveKit docs | Realtime model plugins | LiveKit Docs | Agent session | LiveKit docs

but i want you to help me craft this please, this should be able to be done in someway

Absolutely — I’ll craft a complete, production-ready Python implementation you can drop into a LiveKit AgentSession. It uses the OpenAI Realtime out-of-band approach (

response.create

/

conversation: "none"

) to keep fillers out of history, ensures no overlaps by awaiting playback completion, keeps thinking-sound playback in-between phrases via BackgroundAudioPlayer, runs the long task in the background, and notifies the user when the result is ready. Before I produce the final code, two quick clarifying questions so I match your environment exactly: 1. Do you prefer the server to send raw

response.create

events over the OpenAI realtime websocket (your POC), or do you want a pure AgentSession API variant using

session.say()

/ speech handles (requires text-only Realtime + TTS plugin)? 2. In your POC, how do you detect filler completion today? Do you have a websocket event that sets

_filler_done_event

(server-side) when the filler audio ends on the client, or do you want the server to cancel by response id? Tell me which option (1 or 2) and how filler completion is signalled, and I’ll post a complete, annotated, copy‑paste Python implementation wired to your exact flow.

i'm open to modify the AgentSession API to have a generate_oob_reply

Great — since you're open to adding

generate_oob_reply

to AgentSession, I'll provide a complete, production-ready Python implementation you can drop into your LiveKit agent. It: • Sends out-of-band (OOB) LLM-generated fillers via the OpenAI Realtime

response.create

behavior (conversation: "none") so fillers do not pollute main conversation history. • Ensures fillers never overlap: waits for each filler to finish before backoff and next filler. • Uses BackgroundAudioPlayer for thinking-sound playback between fillers (pauses automatically during speech). • Runs the long tool in background, returns immediately, and notifies when done. • Exposes a small AgentSession helper method

generate_oob_reply(...)

(and a SpeechHandle-like object) so you can integrate cleanly into AgentSession-based code. Below is the implementation broken into: 1) AgentSession extension:

generate_oob_reply

2) Server-side filler manager + long-task tool usage 3) WebSocket handler notes (signals filler playback completion) 1) AgentSession extension (add to your server agent code)

# agent_oob.py
import uuid
import asyncio
from typing import Optional

class OOBSpeechHandle:
    def __init__(self, id: str):
        self.id = id
        self._done = asyncio.Event()
        self._interrupted = False

    async def wait(self, timeout: Optional[float] = None):
        try:
            await asyncio.wait_for(self._done.wait(), timeout=timeout)
            return True
        except asyncio.TimeoutError:
            return False

    def set_done(self):
        self._done.set()

    def interrupt(self):
        self._interrupted = True
        self.set_done()

    @property
    def interrupted(self):
        return self._interrupted

# Monkey-patch or extend AgentSession with generate_oob_reply
def install_generate_oob(session, websocket_send_json):
    """
    session: AgentSession instance
    websocket_send_json: coroutine to send raw realtime websocket payloads to OpenAI
    Adds session.generate_oob_reply(instructions, metadata=None, output_modalities=["audio"]) ->
    returns OOBSpeechHandle
    """

    async def generate_oob_reply(instructions: str, metadata: dict = None, output_modalities=None):
        if output_modalities is None:
            output_modalities = ["audio"]

        fid = str(uuid.uuid4())
        payload = {
            "type": "response.create",
            "response": {
                "conversation": "none",
                "output_modalities": output_modalities,
                "input": [],
                "tool_choice": "none",
                "instructions": instructions,
                "metadata": {**(metadata or {}), "filler_id": fid},
            },
        }

        handle = OOBSpeechHandle(fid)
        # Register handle so external websocket handler can set_done() when playback finishes.
        # store on session so other code can access it
        if not hasattr(session, "_oob_handles"):
            session._oob_handles = {}
        session._oob_handles[fid] = handle

        # send payload over underlying realtime websocket
        await websocket_send_json(payload)
        return handle

    session.generate_oob_reply = generate_oob_reply

    # helper to mark completion from websocket handler
    def _mark_oob_done(filler_id: str):
        h = session._oob_handles.get(filler_id)
        if h:
            h.set_done()
            del session._oob_handles[filler_id]

    def _cancel_oob(filler_id: str):
        h = session._oob_handles.get(filler_id)
        if h:
            h.interrupt()
            del session._oob_handles[filler_id]

    session._mark_oob_done = _mark_oob_done
    session._cancel_oob = _cancel_oob

Notes: •

websocket_send_json

should be your function that sends a JSON payload to the OpenAI realtime websocket (the same socket your plugin uses). • The websocket handler (below) must call

session._mark_oob_done(filler_id)

when a `response.delta`/`response.created`/`response.completed` or client playback-complete event indicates the filler finished; or call

_cancel_oob

if playback aborted. 2) Tool implementation using

generate_oob_reply

+ BackgroundAudioPlayer

# tool_with_oob.py
import asyncio
from livekit.agents import function_tool, RunContext, BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip

BACKOFF_SECONDS = 3.0

@function_tool()
async def long_tool_with_oob(self, context: RunContext, query: str) -> str:
    # ensure BackgroundAudioPlayer running for session (start once per session in real code)
    background_audio = BackgroundAudioPlayer(thinking_sound=[AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.6)])
    await background_audio.start(room=context.room, agent_session=context.session)

    stop_flag = False
    active_handle = None

    async def filler_manager():
        nonlocal active_handle, stop_flag
        stages = ["early", "mid", "late"]
        i = 0
        while not stop_flag:
            # backoff: thinking sound plays during this time
            await asyncio.sleep(BACKOFF_SECONDS)
            if stop_flag:
                break

            stage = stages[min(i, len(stages)-1)]
            instructions = f"Produce one short, natural filler sentence for wait stage: {stage}."

            # send out-of-band filler; get a speech handle
            handle = await context.session.generate_oob_reply(instructions=instructions, metadata={"stage": stage})
            active_handle = handle

            # wait for playback to finish or be interrupted
            finished = await handle.wait(timeout=30.0)  # tune timeout
            active_handle = None
            if not finished:
                # timed out: proceed (or optionally cancel via session._cancel_oob)
                pass

            i += 1

    async def long_work():
        # replace with real long-running logic
        await asyncio.sleep(25)
        return f"Final results for {query}"

    filler_task = asyncio.create_task(filler_manager())
    work_task = asyncio.create_task(long_work())

    # immediate, non-historical audio confirmation
    await context.session.say("I've started processing that. I'll let you know when it's ready.", allow_interruptions=True, add_to_chat_ctx=False)

    async def notifier():
        nonlocal stop_flag, active_handle
        try:
            result = await work_task
            stop_flag = True
            # if filler active, try graceful wait then cancel if needed
            if active_handle:
                await active_handle.wait(timeout=3.0)
                if not active_handle.interrupted:
                    # optionally cancel at realtime side if supported
                    try:
                        context.session._cancel_oob(active_handle.id)
                    except Exception:
                        pass

            # ensure filler manager stops
            filler_task.cancel()
            try:
                await filler_task
            except asyncio.CancelledError:
                pass

            # final reply into history
            await context.session.generate_reply(instructions=f"The result is ready: {result}", allow_interruptions=True)
        except Exception:
            stop_flag = True
            filler_task.cancel()
            await context.session.generate_reply(instructions="There was an error finishing your request.", allow_interruptions=True)

    asyncio.create_task(notifier())

    return "Processing started."

3) WebSocket handler / realtime event wiring • Your OpenAI realtime socket receives events about responses. Wire events to mark OOB filler completion. • Example pseudocode inside your websocket message handler:

# when you see response.completed or playback finished for OOB:
if event.get("type") == "response.completed":
    resp = event["response"]
    # ensure it's a filler (conversation none and metadata.filler_id present)
    fid = resp.get("metadata", {}).get("filler_id")
    if fid:
        # mark done on session so server filler_manager continues
        session._mark_oob_done(fid)

# If client plays audio and sends a data message back when audio finished:
# on data message: if data.type == "filler_finished" and data.filler_id: session._mark_oob_done(data.filler_id)

Client-side option (recommended for absolute reliability): • The client plays the OOB audio and posts a data message back to the server when playback ends: ◦ publish_data({"type":"filler_finished","filler_id": fid}) • Server listens for that data message and calls session._mark_oob_done(fid). Why this meets your requirements • Non-overlap: server awaits

handle.wait()

which only completes when websocket/client signals playback finished — next filler only starts after backoff. • Thinking sound: BackgroundAudioPlayer plays automatically during backoff and pauses during OOB speech. • No conversation pollution:

response.create

uses conversation:"none". • Clean integration: method

session.generate_oob_reply()

mirrors existing session API and returns a handle to await/interrupt. Best practices and tuning • BACKOFF_SECONDS: 2–4s typical. • OOB timeout: short (20–30s) so you'll fallback or cancel if playback stalls. • Use client-side filler_finished events for reliable end-of-playback detection. • If you can, support

response.cancel

on the realtime socket to abort active filler immediately when job completes. • Start BackgroundAudioPlayer once per session (not per tool) in real code. If you want, I’ll: • Provide a variant that uses

session.say()

+ SpeechHandle instead of raw

response.create

(for environments where you prefer AgentSession-only APIs). • Convert the code to a drop-in patch/PR style patch for your AgentSession class. • Add the exact websocket send/receive snippets matching your POC event names. Which next step would you like: (A) full AgentSession-only variant, (B) a PR-style patch for AgentSession with tests, or (C) add client-side filler playback + data-message example?

without altering code this much, can we make this with LiveKit existing codebase?

Yes — you can do this with the existing LiveKit codebase and minimal changes. Use the built-in methods and flow below to get non‑overlapping, out‑of‑band filler phrases, automatic thinking sounds, background tool execution, and clean final notification — without adding new core APIs. Short plan (existing APIs only) • Use BackgroundAudioPlayer for continuous thinking sound (pauses automatically during speech). • Emit fillers with session.say(..., add_to_chat_ctx=False, allow_interruptions=False) so they: ◦ play as audio only (no convo pollution), ◦ are not interrupted by the agent itself. • Ensure non-overlap by awaiting the SpeechHandle returned by session.say (or checking session.current_speech) before scheduling the next filler. • Run the long task as an asyncio background task; stop the filler loop immediately when it finishes (or cancel current speech if needed). • Use session.generate_reply() for the final LLM response (if you want it in history) or session.say/add publish_data for non-historical final notifications. Concise copy-paste pattern (Python asyncio, uses only LiveKit APIs)

import asyncio
from livekit.agents import function_tool, RunContext, BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip

BACKOFF_SECONDS = 3.0  # wait after filler finishes

@function_tool()
async def long_tool_with_fillers(self, context: RunContext, query: str) -> str:
    # start thinking-sound once per session (do this once globally in real code)
    background_audio = BackgroundAudioPlayer(
        thinking_sound=[AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.6)]
    )
    await background_audio.start(room=context.room, agent_session=context.session)

    stop_flag = False
    active_handle = None

    async def filler_loop():
        nonlocal active_handle, stop_flag
        phrases = [
            "Let me check that for you...",
            "Looking that up now...",
            "Still working on it...",
            "Almost done..."
        ]
        i = 0
        while not stop_flag:
            # Wait backoff (thinking sound will play during this)
            await asyncio.sleep(BACKOFF_SECONDS)
            if stop_flag:
                break

            # Prepare filler
            phrase = phrases[i % len(phrases)]

            # Start filler audio via TTS; do not add to chat history and prevent self-interruption
            # `session.say` returns a SpeechHandle-like awaitable in the SDK
            handle = await context.session.say(
                phrase,
                add_to_chat_ctx=False,
                allow_interruptions=False,
            )
            active_handle = handle

            # Wait for playback to finish (prevents overlap). If SDK exposes wait_if_not_interrupted(), use it.
            try:
                await handle  # await completion of playback
            except asyncio.CancelledError:
                # If cancelled, stop handle if available
                try:
                    await handle.stop()
                except Exception:
                    pass
                raise
            finally:
                active_handle = None

            # After playback, backoff again (small granularity to allow prompt stop)
            slept = 0.0
            while slept < BACKOFF_SECONDS and not stop_flag:
                await asyncio.sleep(0.1)
                slept += 0.1

            i += 1

    async def long_work():
        # replace with your long-running work
        await asyncio.sleep(20)
        return f"Results for {query}"

    filler_task = asyncio.create_task(filler_loop())
    work_task = asyncio.create_task(long_work())

    # Immediate non-historical acknowledgement
    await context.session.say(
        "I've started processing that and will tell you when it's ready.",
        add_to_chat_ctx=False,
        allow_interruptions=True,
    )

    async def notifier():
        nonlocal stop_flag, active_handle
        try:
            result = await work_task
            stop_flag = True

            # If a filler is currently playing, wait a short time for it to finish, else cancel it
            if active_handle:
                try:
                    await asyncio.wait_for(active_handle, timeout=3.0)
                except asyncio.TimeoutError:
                    # If SDK supports cancelling active speech, call it:
                    try:
                        await active_handle.stop()
                    except Exception:
                        pass

            # ensure filler loop is stopped
            filler_task.cancel()
            try:
                await filler_task
            except asyncio.CancelledError:
                pass

            # Final response into history (LLM)
            await context.session.generate_reply(
                instructions=f"The result is ready: {result}"
            )
        except Exception:
            stop_flag = True
            filler_task.cancel()
            await context.session.generate_reply(instructions="There was an error finishing your request.")

    asyncio.create_task(notifier())
    return "Processing started."

Notes and small adaptations • session.say returns/behaves like a SpeechHandle you can await; if your SDK names methods differently, replace awaits with the SDK’s speech-handle API (e.g., wait_if_not_interrupted()). • BackgroundAudioPlayer automatically pauses during speech and resumes after — no manual toggling required. • Use add_to_chat_ctx=False to avoid polluting history. Use allow_interruptions=False to avoid self-overlap; set True if you want user interruptions to cut fillers short. • If you need even stronger guarantees or client-side exact playback detection, have the client publish a data message when filler audio finishes and check that event before continuing — but this is optional. If you want, I can: • Produce a minimal variant that uses client-side data messages for exact end-of-playback signaling. • Convert this into a small helper function you can reuse across tools (no framework change). Which would you like next? Sources: Agent speech and audio | LiveKit docs | Agent session | LiveKit docs | External data and RAG | LiveKit docs

but the STT has to speak the same as the model, otherwise it will be weird for the user

You're right — voice mismatch is jarring. You can achieve consistent voice without major changes by ensuring the same TTS voice is used for both model-driven speech and any filler/TTS you emit. Two practical, minimal-change options: Option A — Realtime model for comprehension + same TTS for all agent speech (recommended) • Configure AgentSession with the OpenAI Realtime model in text modality and a TTS plugin using the same voice you want the model to speak in. ◦ llm = openai.realtime.RealtimeModel(modalities=["text"]) ◦ tts = cartesia/elevenlabs/your_tts(voice="desired_voice") • Use: ◦ context.session.generate_reply(...) for LLM final replies (in-history). ◦ context.session.say(..., add_to_chat_ctx=False, allow_interruptions=...) for out-of-band fillers (TTS, not added to history). • BackgroundAudioPlayer handles thinking sounds and will pause automatically when TTS plays. • This makes all spoken audio use the same configured TTS voice so the user hears a single consistent voice. Option B — Use Realtime model's audio + synth fillers via matching TTS (if you must use Realtime audio) • If you must let the Realtime model synthesize some responses directly, pick a TTS voice that matches it as closely as possible and use session.say(...) with that TTS for fillers. • Prefer client-side playback signaling (publish_data) to ensure filler completion synchronization. Practical checklist / minimal code hints • Set AgentSession lls/tts once at session start so both generate_reply and say use the same TTS. • Emit fillers with: await context.session.say("…", add_to_chat_ctx=False, allow_interruptions=False) await handle # wait to avoid overlap • Use BackgroundAudioPlayer for typing/thinking sound; it auto-pauses during speech. • Final result: use generate_reply(...) so the answer is in history and spoken with the same voice. If you want, I’ll produce a very small, concrete snippet that shows AgentSession creation with the matching TTS voice and a short tool that uses

session.say()

fillers plus

generate_reply()

final output. Which TTS provider/voice are you using (Cartesia, ElevenLabs, or other)?

let's go back to the custom implementation, i want to make this integrated into the existing LiveKit flow (so it matches agent activity state transitions and stuff) but seems like a not well handed case right?

You're right — this is a real gap: LiveKit supports the building blocks (speech handles, thinking sounds, session.say, agent state events), but there isn't a single built-in shortcut that cleanly implements non‑overlapping, out‑of‑band LLM-generated fillers tied into the agent lifecycle. You can solve it two ways: 1) Minimal-change, fully-supported approach (recommended) • Use the existing APIs only: ◦ Configure the session with the same TTS used for all agent speech (Realtime as text + TTS) so voice matches. ◦ Play thinking sound via BackgroundAudioPlayer. ◦ Emit fillers with session.say(..., add_to_chat_ctx=False, allow_interruptions=False). Await the returned SpeechHandle (or check session.current_speech) before scheduling the next filler to guarantee no overlap. ◦ Run the long tool as an asyncio task and stop the filler loop when it completes. Use session.generate_reply() for the final answer. • Tradeoffs: no core changes, reliable voice consistency, fillers are TTS (not LLM-generated). Works inside all LiveKit agent state transitions. 2) Custom integration for LLM-generated OOB fillers (more control) • Add a small helper on top of AgentSession (no deep core change) that sends OpenAI realtime

response.create

with

"conversation": "none"

and returns an OOB SpeechHandle. This keeps fillers out of history and lets the model author the phrase. • Wire the realtime websocket events (or client playback data messages) to mark OOB playback completion so the handle can be awaited. Use BackgroundAudioPlayer for thinking sound between fills. • Tradeoffs: more plumbing (websocket wiring or client signaling) but you get LLM-generated fillers without polluting history and exact non-overlap control. Practical recommendation • If matching voice and minimal maintenance are most important: go with option 1. • If you need LLM-authored, natural fillers that must not appear in history: use option 2 (the helper/handle approach you and I sketched earlier). It’s implementable without changing core LiveKit — just add a small helper module and hook websocket or client data events to mark playback completion. I can: • Deliver a compact, production-ready drop-in helper for option 2 (AgentSession wrapper + websocket handler + tool example) that integrates with agent state and BackgroundAudioPlayer, or • Produce the minimal option-1 snippet tuned for your TTS provider so it uses only existing APIs. Which do you want me to produce now: (A) the full drop-in helper for LLM OOB fillers (response.create + handle + wiring), or (B) the minimal no-core-change snippet that uses session.say + BackgroundAudioPlayer?