https://livekit.io logo
Join Slack
Powered by
How to solve uncaught exception when user exit roo...
# ask-ai
v
How to solve uncaught exception when user exit room? here is agent log: Received disconnect RPC; shutting down agent session [074015.664] INFO (9491): mainTask: draining and no more speech tasks end_turn ignored because no turn is active [074015.665] INFO (9491): AgentActivity mainTask: exiting [074015.669] INFO (9491): AgentSession closed reason: "user_initiated" error: null [074017.134] INFO (9491): Session report uploaded to LiveKit Cloud libc++abi: terminating due to uncaught exception of type std: 1system error mutex lock failed: Invalid argument
t
Thanks for reaching out, Dức. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — you're asking how to fix the uncaught C++ exception shown in your log when a user exits a room. I’ve reviewed the log lines you posted and below are targeted next steps to diagnose and fix it. What the log shows • The agent session cleanly shuts down (draining, exiting, closed, report uploaded). • A few seconds later the process aborts with: libc++abi: terminating due to uncaught exception of type std: 1system error mutex lock failed: Invalid argument • This pattern points to a post-shutdown race or a resource cleanup problem: a background thread/task tries to lock or access a mutex that has already been destroyed. What you can do next (ordered by easiest to most diagnostic) 1. Confirm versions • Note the exact versions of 
livekit-agents
, 
livekit-server-sdk
, and any native plugins or inference libraries you use. If you can, upgrade to the latest release of those packages first. 1. Inspect shutdown ordering and hooks • Ensure all shutdown hooks and cleanup callbacks run deterministically and that long-running async tasks are canceled or awaited before final teardown. • If you register shutdown callbacks (e.g., via 
ctx.add_shutdown_callback
), make them cancel background workers or await their completion. 1. Make background tasks cancelable • Any background thread, worker, or plugin task that might still run after session close should be made cancelable and explicitly stopped during shutdown. • For async tasks, call cancel and await them to finish rather than leaving them to run after teardown. 1. Check native/plugin modules • If you use native extensions (model inference engines, C++ helpers, or other plugins), inspect their cleanup: background threads inside native modules commonly cause this error when they outlive associated mutexes. • For any plugin that supports a "stop" or "destroy" API, call that during session shutdown and confirm it returns before process exit. 1. Add logging and reproduce • Add logging around shutdown sequence and plugin cleanup to identify which component continues to run after AgentSession closed. • Try to reproduce consistently with a controlled user disconnect; capture a full native stack trace if possible (run under a debugger or enable core dumps) to see which module raises the exception. 1. Temporary mitigation • If you can’t immediately fix the underlying module, ensure the process is restarted automatically after crash to restore service while you debug. If you’d like, provide: • Versions of 
livekit-agents
, 
livekit-server-sdk
, and any plugins/native deps. • Whether you use any plugins that prewarm or run background workers (name them). • Any custom native/C++ modules or third-party inference libraries in use. I can then suggest more specific changes (where to add cancellation, what to await in shutdown hooks, or which plugin cleanup to inspect). Would you like to share the versions and plugin list now? Sources: Job lifecycle | LiveKit docs | Agent server options | LiveKit docs
v
"livekit-server-sdk":"^2.15.0" "@livekit/agents": "^1.0.21",
t
Thanks — you shared your versions (
livekit-server-sdk
^2.15.0 and 
@livekit/agents
^1.0.21). That helps. Next targeted steps to narrow and fix the mutex crash: 1. Reproduce with verbose shutdown logs • Add logs immediately before and after every shutdown step (drain, stop workers, plugin stop, session close, report upload) so we can see which action finishes just before the crash. 1. Confirm plugins/native deps • Do you use any plugins or native/compiled libraries (e.g., model inference engines, prewarm plugins, or C++ addons)? Name them and their versions. Native modules are the most likely culprits. 1. Ensure all background workers are stopped before exit • In your shutdown sequence, explicitly: ◦ Cancel and await all async tasks. ◦ Stop any thread pools or native worker threads via their stop/destroy APIs. ◦ Call plugin cleanup functions (if present) and wait for completion. • If you use 
ctx.add_shutdown_callback
, make the callback synchronous or ensure it awaits worker shutdown before returning. 1. Test isolations • Run the agent with plugins disabled (if possible). If crash stops, enable plugins one-by-one to find the offender. • If you have custom native code, run a build with debug symbols or run under a debugger to capture a native stack trace when it crashes. 1. Collect a native stack trace / core dump • If the crash persists, capture a native stack trace (gdb/lldb) or core dump at crash time. That stack will show which mutex and module raised the error and points to the exact cleanup race. If you want, share: • The list of plugins/native deps and their versions. • The shutdown log (with added timestamps around each shutdown step). • Whether disabling plugins prevents the crash. I’ll use that to point to the exact shutdown step or plugin to fix and give code-level suggestions for cancellation/ordering.
v
prewarm: async (proc*:* JobProcess) => { proc.userData.vad = await silero.VAD.load(); }, "@livekit/agents-plugin-livekit": "^1.0.21", "@livekit/agents-plugin-openai": "^1.0.27", "@livekit/agents-plugin-silero": "^1.0.21", "@livekit/noise-cancellation-node": "^0.1.9",
t
Thanks — this is very helpful. You're using 
@livekit/agents
^1.0.21 with plugins including 
agents-plugin-silero
and you prewarm VAD with an async call that stores a native resource on 
proc.userData.vad
. That strongly suggests the crash is caused by a native/background thread from the Silero VAD (or another native plugin like noise-cancellation) continuing to run after session teardown and attempting to lock a destroyed mutex. What to try next (concise, ordered): 1. Stop/await VAD and plugin workers during shutdown • On session shutdown, explicitly call the VAD/plugin cleanup API (if available) and await it before finishing shutdown. Example pattern: a. Cancel background tasks. b. Call 
await proc.userData.vad?.stop()
or the plugin's destroy/close method. c. Then close the AgentSession and finish other cleanup. 1. Make prewarm resources cancellable and reference-counted • Instead of leaving native handles on 
proc.userData
without teardown, track them and ensure they are destroyed when the process/session ends. If multiple sessions share a prewarmed instance, use a refcount and only destroy when last user leaves. 1. Isolate and reproduce by disabling plugins • Run the agent with 
agents-plugin-silero
and 
noise-cancellation-node
disabled. If the crash stops, re-enable one plugin at a time to identify the offender. 1. Add targeted shutdown logs • Log before/after: plugin stop, VAD stop, noise-cancel stop, AgentSession close, report upload. That will show which stop step completes just before the crash. 1. Capture native stack trace if still crashing • If the crash persists after explicit plugin shutdown, capture a native stack trace (lldb/gdb or core dump) to see which library/mutex threw the exception. If you want, I can provide: • Example shutdown snippet showing how to cancel tasks and await plugin/vad stop. • A short checklist for refcounting shared prewarmed native handles. Which would you like next — a code snippet for shutdown cleanup or a checklist for isolating the plugin?
v
How to run the agent with 
agents-plugin-silero
and 
noise-cancellation-node
disabled?
t
You're already close — to run the agent with 
@livekit/agents-plugin-silero
and 
@livekit/noise-cancellation-node
disabled, stop loading or passing them into the AgentSession and do not enable noise cancellation in 
inputOptions
. Concretely: 1. Remove any imports and the prewarm VAD load:
Copy code
// REMOVE or comment out
// import * as silero from '@livekit/agents-plugin-silero';
// proc.userData.vad = await silero.VAD.load();
1. Start the session without a 
vad
field and without noise cancellation:
Copy code
await session.start({
  room: ctx.room,
  agent,
  inputOptions: {
    audioEnabled: true,
    textEnabled: true,
    // do NOT include noiseCancellation or BackgroundVoiceCancellation()
  },
  // do NOT set turnDetection to a silero-dependent mode
});
1. (Optional) Uninstall the packages if you want them completely absent: pnpm remove @livekit/agents-plugin-silero @livekit/noise-cancellation-node Notes: • Without VAD, use 
turnDetection: 'stt'
or 
'manual'
if needed. • This isolates native plugin behavior so you can see if the crash stops when those plugins are disabled. Would you like a ready-to-run minimal agent file (complete example) to test this? Sources: Voice AI quickstart | LiveKit docs | Enhanced noise cancellation | LiveKit docs | Silero VAD plugin | LiveKit docs | Turn detection and interruptions | LiveKit Docs | Noise & echo cancellation | LiveKit docs
v
is this correct shutdown agent handling? localParticipant?.registerRpcMethod('closeSession', async () => { console.log('Received disconnect RPC; shutting down agent session'); await session.close(); ctx.shutdown?.('client_requested_logout'); return 'ok'; });
t
You're on the right track — the RPC handler mostly works, but calling both 
session.close()
and 
ctx.shutdown()
can cause redundant/conflicting teardown and may contribute to post-shutdown races. Use one clear shutdown path. Recommended change:
Copy code
localParticipant?.registerRpcMethod('closeSession', async () => {
  console.log('Received disconnect RPC; shutting down agent session');
  await session.close(); // performs session cleanup and disconnect
  return 'ok';
});
If instead you want the JobContext to drive shutdown, call only 
ctx.shutdown?.('client_requested_logout')
and don't call 
session.close()
. Also ensure before closing: • Cancel and await any background tasks (workers, timers). • Explicitly stop/await plugin native resources (e.g., 
await proc.userData.vad?.stop()
). • Remove or await any fire-and-forget tasks started during the session. Would you like a short shutdown helper snippet that cancels tasks and stops plugin resources before calling 
session.close()
? Sources: Job lifecycle | LiveKit docs | Agent session | LiveKit docs
v
yes
t
Below is a concise shutdown helper you can add and call from your RPC handler. It cancels tracked background tasks, stops plugin/native resources (VAD and noise cancellation), then awaits 
session.close()
.
Copy code
// keep a Set of running tasks
const runningTasks = new Set<Promise<any>>();

function trackTask(p: Promise<any>) {
  runningTasks.add(p);
  p.finally(() => runningTasks.delete(p));
  return p;
}

// Example: when creating background tasks, wrap them:
// trackTask(someLongRunningTask());

// shutdown helper
async function gracefulShutdown(proc: any, session: any) {
  console.log('gracefulShutdown: cancelling background tasks');

  // 1) Cancel/await tracked tasks
  // If tasks are cancelable via AbortController, signal them here.
  // For plain promises, we await them with a timeout to avoid hanging forever.
  const TIMEOUT = 5000;
  try {
    await Promise.race([
      Promise.allSettled(Array.from(runningTasks)),
      new Promise((_, rej) => setTimeout(() => rej(new Error('tasks timeout')), TIMEOUT)),
    ]);
  } catch (err) {
    console.warn('Some background tasks did not finish before shutdown:', err);
  }

  // 2) Stop plugin/native resources stored on proc.userData
  try {
    if (proc?.userData?.vad?.stop) {
      console.log('Stopping VAD');
      await proc.userData.vad.stop();
    }
  } catch (err) {
    console.warn('Error stopping VAD:', err);
  }

  try {
    if (proc?.userData?.noiseCancellation?.stop) {
      console.log('Stopping noise cancellation');
      await proc.userData.noiseCancellation.stop();
    }
  } catch (err) {
    console.warn('Error stopping noise cancellation:', err);
  }

  // 3) Give native modules a moment to tear down (optional small delay)
  await new Promise((res) => setTimeout(res, 100));

  // 4) Close the AgentSession (single authoritative shutdown)
  try {
    console.log('Closing AgentSession');
    await session.close();
  } catch (err) {
    console.warn('Error during session.close():', err);
  }
}
Then call it from your RPC:
Copy code
localParticipant?.registerRpcMethod('closeSession', async () => {
  console.log('Received disconnect RPC; shutting down agent session');
  await gracefulShutdown(proc, session);
  return 'ok';
});
Notes: • Replace 
proc.userData.vad.stop()
/ 
proc.userData.noiseCancellation.stop()
with the actual cleanup API your plugins expose if different. • Prefer cancelable tasks (AbortController) for background work and call their cancel method before awaiting. • Keep only one shutdown path: call 
session.close()
(not 
ctx.shutdown()
afterwards). • If crash persists, run the agent with those plugins disabled to confirm the native plugin is the cause.
v
will agent leave room after shutting down ?
t
Yes — when you call 
session.close()
(or 
ctx.shutdown()
as the single shutdown path), the agent will leave the room and stop participating. Key points: • 
session.close()
performs graceful disconnect: drains queued operations, closes I/O, runs shutdown hooks, and disconnects the agent from the room. • Other participants remain in the room unless you explicitly delete the room via your server-side API. • Use a single shutdown path (prefer 
session.close()
), and ensure you stop/await plugin/native resources before calling it so no background threads outlive teardown. Would you like a one-line confirmation snippet to place in your RPC handler that calls the gracefulShutdown helper and then closes the session? Sources: Job lifecycle | LiveKit docs
v
how to explicitly delete the room via your server-side API ?
t
You can delete a LiveKit room from your server using the RoomService/DeleteRoom API. Deleting the room disconnects all participants and closes it. Below is a concise Node.js example and notes. Node.js (livekit-server-sdk)
Copy code
import { RoomServiceClient } from 'livekit-server-sdk';

const livekitHost = '<https://your-project.livekit.cloud>'; // or your LiveKit server URL
const apiKey = process.env.LIVEKIT_API_KEY;
const apiSecret = process.env.LIVEKIT_API_SECRET;

const roomService = new RoomServiceClient(livekitHost, apiKey, apiSecret);

// delete room by name (case-sensitive)
await roomService.deleteRoom('my-room-name');
Python (async)
Copy code
from livekit.api import LiveKitAPI, DeleteRoomRequest

async with LiveKitAPI("<https://your-project.livekit.cloud>", "api-key", "secret") as api:
    await api.room.delete_room(DeleteRoomRequest(room="my-room-name"))
Go
Copy code
roomClient := lksdk.NewRoomServiceClient("<https://your-project.livekit.cloud>", "api-key", "secret")
_, _ = roomClient.DeleteRoom(context.Background(), &livekit.DeleteRoomRequest{
  Room: "my-room-name",
})
Notes • Use your server API key/secret (do not expose on clients). • The argument is the room name (case-sensitive). • Call deleteRoom after any final cleanup you need (e.g., finalize recordings). • Deleting a room immediately disconnects all participants. Would you like a small snippet you can drop into your existing shutdown flow to call deleteRoom after gracefulShutdown? Sources: Managing rooms | LiveKit docs | server-sdk-js | livekit/livekit_composite | DeepWiki | Job lifecycle | LiveKit docs | server-sdk-go | livekit/livekit_composite | DeepWiki
v
is that ok to delete room without calling rpc to shut down agent ?
t
You can delete the room without first calling an RPC to shut down the agent — deleteRoom will force-disconnect all participants, including the agent. However, best practice is to gracefully stop the agent first (cancel background tasks, stop native/plugin resources, then 
session.close()
), then call deleteRoom if you need to permanently close the room. Reasons: • Graceful shutdown ensures transcripts, uploads, and plugin cleanup run and reduces risk of native race conditions (mutex errors). • deleteRoom is immediate and may interrupt cleanup, which can leave native resources running long enough to crash the process. If you need a simple flow: 1. gracefulShutdown(proc, session) // cancel tasks, stop plugins, await session.close() 2. call RoomServiceClient.deleteRoom(roomName) If you prefer a single-step forced close (and accept the risks), calling deleteRoom alone will disconnect the agent. Which flow would you prefer to implement?
v
in push to talk agent, do client need to call rpc to agent to setAudioEnabled(true/false) since client handled room.localParticipant.publishTrack and room.localParticipant.unpublishTrack already ?
t
Short answer: Yes — the client should still call an RPC to the agent to toggle 
session.input.setAudioEnabled(true/false)
even if it publishes/unpublishes its audio track. Why: • publish/unpublish controls whether audio is sent to the room. • 
setAudioEnabled
controls whether the agent processes that incoming audio (STT/turn detection). If the agent's input is disabled it will ignore audio even if the track is published. • Calling the RPC provides a clear signal of user intent (start/stop speaking), helps with turn-taking, avoids partial/garbage input, and lets the agent cancel or buffer work appropriately. Best practice: • Client publishes track when user presses push-to-talk. • Client calls RPC to enable agent input at the same time. • On release, client unpublishes track and calls RPC to disable agent input (or the agent can wait for STT endpointing if you prefer). • Ensure agent-side logic cancels/awaits processing when input is disabled to avoid race conditions. If you want, I can provide a minimal client+agent RPC sequence snippet for push-to-talk. Sources: Turn detection and interruptions | LiveKit Docs
v
after end turn, the event RoomEvent.TranscriptionReceived doesn't trigger again when start new turn (was triggered on first turn) const handleEndTurn = useCallback( async (room: Room) => { try { const agentIdentity = getAgentIdentity(room); if (!agentIdentity) { throw new Error("No agent participant found for RPC"); } if (trackRef.current) { await room.localParticipant.unpublishTrack(trackRef.current); await trackRef.current.mute() } await room.localParticipant.performRpc({ destinationIdentity: agentIdentity, method: "endTurn", payload: "{}" }); } catch (error) { throw error; } }, [getAgentIdentity] );
t
You're unpublishing and ending the turn, so the agent stops listening. For a new turn you must both restore audio publishing and tell the agent to start listening again — otherwise transcription events won't be produced. What to do (minimal): 1. When starting a new turn: ◦ publish the audio track (or ensure it's published) ◦ call the agent RPC to enable input / start the turn (e.g., 
performRpc
to call 
startTurn
or 
session.input.setAudioEnabled(true)
) 1. When ending a turn (what you already do): ◦ unpublish or mute the track ◦ call 
endTurn
/ 
session.input.setAudioEnabled(false)
Example flow (client): • On push-to-talk press: a. await room.localParticipant.publishTrack(track) b. await room.localParticipant.performRpc({ destinationIdentity: agentIdentity, method: "startTurn", payload: "{}" }) • On release: a. await room.localParticipant.unpublishTrack(track) b. await room.localParticipant.performRpc({ destinationIdentity: agentIdentity, method: "endTurn", payload: "{}" }) Also verify on the agent side: • 
startTurn
enables audio input and resets any per-turn transcription state. • 
endTurn
stops listening and finalizes any buffered transcription. • Ensure the agent subscribes to transcription events each turn (no one-time listeners that are removed on first turn). If you want, share your start-turn client code and the agent's `startTurn`/`endTurn` handlers and I’ll point out exactly what to change.
v
on push to talk, does client need to createLocalAudioTrack per turn ?
t
You do not need to create a new LocalAudioTrack for every push‑to‑talk turn. Reusing one track is simpler and recommended in most cases. Why reuse is best • Lower latency and CPU: creating tracks (and acquiring mic permissions) per turn adds delay. • Fewer resources: one track avoids repeated device reinitialization. • Simpler lifecycle: publish/unpublish or mute/unmute a single track instead of creating/destroying. Two common patterns 1) Reuse single track (recommended) • Create the LocalAudioTrack once when the user joins or first uses push-to-talk. • On push: publish or unmute the track and notify the agent (RPC) to start listening. • On release: unpublish or mute the track and notify the agent to stop listening. Example flow: • init: track = await createLocalAudioTrack() • push: await localParticipant.publishTrack(track); RPC: startTurn • release: await localParticipant.unpublishTrack(track) OR await track.mute(); RPC: endTurn 2) Create per-turn (only if necessary) • Create a new track each press, publish it, then unpublish and stop it on release. • Use this only if you must change device constraints per turn or isolate audio contexts. • Drawback: added permission/device latency and more resource churn. Extra tips • Prefer mute/unmute if you want instant toggles without republishing; publish once and toggle 
track.muted
or call `track.mute()`/`track.unmute()` depending on your SDK. • If you unpublish but keep the track object, you can republish the same track next turn (no need to recreate). • Always coordinate with the agent: publish/unpublish (or mute/unmute) + RPC 
setAudioEnabled(true/false)
or `startTurn`/`endTurn` for correct agent-side processing. If you want, I can provide a minimal client code snippet showing reuse with publish/unpublish and the corresponding RPC calls.
v
diff publish/unpublish vs mute/unmute ?
t
Quick comparison: publish/unpublish vs mute/unmute • What they do ◦ publish/unpublish: Add or remove the track from the Room’s signaling (the track is sent to server/participants only while published). ◦ mute/unmute: Stop or resume sending media on an already-published track without changing the track’s published state. • Signaling & visibility ◦ publish/unpublish: Participants see track published/unpublished events; subscribers may be subscribed/unsubscribed. ◦ mute/unmute: Track remains published; subscribers still see the track but its media is silent/paused. • Bandwidth & codecs ◦ publish/unpublish: When unpublished, no media is sent. On publish, negotiation may occur. ◦ mute/unmute: Media pause typically reduces bandwidth; encoding/negotiation stays intact (lower latency to resume). • Latency to resume sending ◦ publish/unpublish: Higher — may require re-negotiation or re-subscription depending on SDK and server. ◦ mute/unmute: Lower — immediate resume of media flow. • Resource & device handling ◦ publish/unpublish: Can keep or discard the underlying MediaStreamTrack; unpublishing then stopping the track fully releases device resources. ◦ mute/unmute: Keeps device and encoder active. • Agent-side behavior (important for push-to-talk) ◦ publish/unpublish: Agent may stop receiving/subscribing to the track; you must also RPC the agent to enable/disable input processing. ◦ mute/unmute: Agent stays subscribed; agent will receive empty/silent audio — still may need RPC to control STT/turn handling. • Recommended for push-to-talk ◦ Preferred: reuse a single LocalAudioTrack and use publish once + mute/unmute (or publish/unpublish once and then reuse by republishing) depending on your UX: ▪︎ If you need instant resume and minimal latency: publish once and mute/unmute. ▪︎ If you need to fully release mic between turns: unpublish and stop the track (or unpublish and keep the track for republish). ◦ Always pair client-side publish/mute actions with an RPC to the agent to call 
session.input.setAudioEnabled()
or `startTurn`/`endTurn` so the agent knows when to process audio. Example patterns • Low-latency reuse (recommended): ◦ create track once, publish it, then on PTT press call RPC start and unmute; on release call RPC end and mute. • Release resources: ◦ on PTT press create & publish track; on release unpublish and stop track. If you want, I can give a minimal client snippet for your preferred pattern (mute/unmute or publish/unpublish). Which one would you like?
v
how to receive dispatched metadata in handler entry: async (ctx*:* JobContext) of agent ?
t
Access the dispatched metadata from 
ctx.job.metadata
inside your agent entry. It's a string — parse it if you sent JSON. Node.js example:
Copy code
export default defineAgent({
  entry: async (ctx: JobContext) => {
    const metadata = ctx.job.metadata; // string
    if (metadata) {
      try {
        const data = JSON.parse(metadata);
        // use data fields
      } catch {
        console.warn('metadata is not JSON:', metadata);
      }
    }
  },
});
Python example:
Copy code
async def entry(ctx: JobContext):
    metadata = ctx.job.metadata  # string
    if metadata:
        try:
            data = json.loads(metadata)
            # use data fields
        except json.JSONDecodeError:
            print('metadata is not JSON:', metadata)
If you send metadata via API or token, confirm the exact string sent and I can suggest parsing/validation helpers. Sources: Job lifecycle | LiveKit docs | Agent dispatch | LiveKit docs | SIP dispatch rule | LiveKit docs
v
when agent receive response from llm (and audio output of agent is disabled), will agent auto send this llm response to client ?
t
Short answer: Yes — if audio output is disabled the agent will not publish audio, but it will still deliver the LLM response as text via the transcription stream (lk.transcription) if text output is enabled. The raw LLM payload is not automatically forwarded; if you need that you must send it explicitly (for example via an RPC or a custom data message). Sources: Text and transcriptions | LiveKit docs
v
how to receive this LLM response from client ?
t
You can receive the agent’s text (LLM) responses on the client by listening for the transcription text stream the agent publishes: topic "lk.transcription". Register a text stream handler and parse the payload (it's JSON). Minimal example (JS/TS):
Copy code
// assume `room` is an active LiveKit Room

// handler receives: payload (string), participant identity, topic
room.registerTextStreamHandler('lk.transcription', (payload, participant, topic) => {
  try {
    const msg = JSON.parse(payload);
    // common fields: msg.text (the transcript), msg.type, msg.metadata...
    console.log('Agent text response:', msg.text);
    // update UI with msg.text
  } catch (err) {
    console.warn('Non-JSON transcription payload:', payload);
  }
});
Notes: • The transcription for agent output may not include 
lk.transcribed_track_id
when audio output is disabled. • If you need the raw LLM response (beyond the published transcription), have the agent send it explicitly via an RPC or a custom data message; transcription is the standard way to get the agent’s text output.
v
does lk.transcription send to client after this event on agent "voice.AgentSessionEventTypes.ConversationItemAdded" ?
t
Yes — the agent emits ConversationItemAdded when a reply is committed, and the transcription text stream (
lk.transcription
) is published to the room afterward. In other words: ConversationItemAdded → agent output pipeline runs → 
lk.transcription
is sent (synchronized with speech if enabled). What this means for your client: • Listen for 
lk.transcription
to receive the agent’s text responses. • ConversationItemAdded is an agent-side event you can use for internal logic; the client receives the transcription stream after that event completes. • If you need to react earlier or get raw LLM data, have the agent send an RPC or custom data message at ConversationItemAdded time. If you want, I can show a minimal agent-side hook that publishes the transcription or sends an RPC to clients right when ConversationItemAdded fires. Sources: Events and error handling | LiveKit docs | Text and transcriptions | LiveKit docs | Pipeline nodes and hooks | LiveKit docs
v
give me docs link of this "ConversationItemAdded → agent output pipeline runs → 
lk.transcription
is sent (synchronized with speech if enabled)."
t
Here are the official docs that show the sequence you asked about: • ConversationItemAdded event: https://docs.livekit.io/agents/build/events/#conversation_item_added • Pipeline/TTS node (agent output pipeline): https://docs.livekit.io/agents/build/nodes/#tts_node • Transcription forwarding / synchronized transcription (lk.transcription): https://docs.livekit.io/agents/build/text/#synchronized-transcription-forwarding These pages together confirm: ConversationItemAdded is emitted when a reply is committed, the agent output pipeline (TTS) runs next, and the agent publishes the transcription to the lk.transcription stream (synchronized with speech when enabled). Sources: Events and error handling | LiveKit docs | Pipeline nodes and hooks | LiveKit docs | Text and transcriptions | LiveKit docs
v
are there any way to send the llm response to client even before ConversationItemAdded event ?
t
Yes — you can deliver LLM output to the client before 
ConversationItemAdded
by intercepting the LLM/tts pipeline and publishing the text yourself (data channel) while the pipeline continues. Two common hooks: 1) Intercept streaming LLM output (earliest) • Override 
llm_node
(or wrap the llm streaming code) and publish each chunk as it arrives. • Publish via participant data channel (custom topic), e.g. 
room.local_participant.publish_data(chunk.text, topic='lk.llm_raw_output')
. • Continue yielding chunks so the normal pipeline still commits the item and runs TTS. 2) Intercept final text just before TTS • Override 
tts_node
and publish the assembled text (partial or final) before synthesizing. • Useful if you want the final cleaned text but still want TTS sync behavior. Notes: • Use a distinct data topic (not 
lk.transcription
) and have the client register a text stream / data handler for that topic. • This does not bypass the pipeline — it simply forwards text earlier; 
conversation_item_added
will still fire later when the item is committed. • Be mindful of duplicate UI updates (early text vs. later transcription); include metadata (e.g., partial/final) in your messages. If you want, tell me which language (Node/Python) you use and I’ll provide a minimal code snippet for the appropriate hook. Sources: Events and error handling | LiveKit docs | Pipeline nodes and hooks | LiveKit docs | Remote method calls | LiveKit docs
v
show me code to Intercept streaming LLM output (earliest)
t
Below is a concise, ready-to-use Node.js example showing how to intercept streaming LLM output as early as possible by overriding the agent's 
llm_node
. It publishes partial LLM chunks to a custom data topic (
lk.llm.partial
) so clients receive tokens before 
conversation_item_added
. Agent (Node.js / TypeScript)
Copy code
import {
  defineAgent,
  voice,
  llm,
  type JobContext,
} from '@livekit/agents';

// Custom agent class that intercepts llm_node streaming output
class StreamingLLMAgent extends voice.Agent {
  // Note: method name follows SDK casing; adapt if your SDK differs
  async llm_node(
    chatCtx: llm.ChatContext,
    tools: llm.ToolContext,
    modelSettings: voice.ModelSettings,
  ): Promise<AsyncIterable<llm.ChatChunk> | null> {
    const upstream = await voice.Agent.default.llm_node(this, chatCtx, tools, modelSettings);
    if (!upstream) return null;

    // Create an async generator that forwards chunks and publishes them early
    async function* intercept() {
      for await (const chunk of upstream) {
        // Extract text from chunk (supports delta/content shapes)
        let text = '';
        if (typeof chunk === 'string') {
          text = chunk;
        } else if (chunk?.delta?.content) {
          text = chunk.delta.content;
        } else if (chunk?.text) {
          text = chunk.text;
        }

        if (text) {
          try {
            // publish partial text immediately to a custom topic
            // use localParticipant.publishData for small payloads (text)
            // Buffer is required by publishData
            await (this.session?.room?.localParticipant?.publishData(
              Buffer.from(text),
              { topic: 'lk.llm.partial' },
            ));
          } catch (err) {
            console.warn('publishData failed for partial LLM chunk:', err);
          }
        }

        yield chunk; // continue normal pipeline
      }
    }

    return intercept.call(this);
  }
}

export default defineAgent({
  entry: async (ctx: JobContext) => {
    const agent = new StreamingLLMAgent({
      instructions: 'You are a helpful assistant.',
    });

    const session = new voice.AgentSession({
      // configure stt/llm/tts as needed, e.g.:
      // llm: new SomeLLMProvider(...),
      // tts: new SomeTTSProvider(...),
    });

    await session.start({
      room: ctx.room,
      agent,
      inputOptions: { audioEnabled: true, textEnabled: true },
    });

    await ctx.connect();
  },
});
Client (TS/JS) — receive partial tokens
Copy code
// Listen for data_received and handle custom topic
room.on('data_received', (payload, participant) => {
  if (payload.topic === 'lk.llm.partial') {
    const text = new TextDecoder().decode(payload.data);
    // append to UI as partial token
    console.log('LLM partial:', text);
  }
});
Notes and best practices • Use a custom topic (e.g., 
lk.llm.partial
) to avoid colliding with built-in streams. • Throttle or buffer if chunks are very frequent to avoid frontend flood. • Include metadata (partial vs final) if you need to distinguish streaming tokens from the final committed item. • Keep yielding chunks so the pipeline still commits the final item and runs TTS; this only forwards early data. • If you prefer the final pre-TTS text, intercept 
tts_node
instead. If you want the Python version or a variant that includes structured JSON messages (e.g., { partial:true, text:"..." }), tell me which format you prefer and I’ll provide it. Sources: Pipeline nodes and hooks | LiveKit docs | Text and transcriptions | LiveKit docs | Events and error handling | LiveKit docs
v
using nodejs, "@livekit/agents": "^1.0.21", override async llmNode didn't work
t
Thanks — let's narrow this down so we can get a working interception on v1.0.21. Quick checklist first (most common causes) 1. Method name / casing: use 
llm_node
(snake_case) in this SDK version, not 
llmNode
. 2. Signature: it must return an AsyncIterable (async generator) of chunks, not a Stream/Transform. 3. Ensure your custom Agent class is actually used when creating the session (pass instance to session.start). 4. Confirm your code runs in the agent entry (after ctx.connect / before session.start is complete as appropriate). Working minimal example (Node.js, v1.x) — override 
llm_node
and publish partial chunks:
Copy code
import { defineAgent, voice, llm, type JobContext } from '@livekit/agents';

class StreamingAgent extends voice.Agent {
  async *llm_node(
    chatCtx: llm.ChatContext,
    tools: llm.ToolContext,
    modelSettings: voice.ModelSettings,
  ): AsyncIterable<any> {
    const upstream = await (voice.Agent as any).default.llm_node.call(this, chatCtx, tools, modelSettings);
    if (!upstream) return;

    for await (const chunk of upstream) {
      // extract partial text
      let text = '';
      if (typeof chunk === 'string') text = chunk;
      else if (chunk?.delta?.content) text = chunk.delta.content;
      else if (chunk?.text) text = chunk.text;

      if (text) {
        try {
          // publish partial text to custom topic
          await this.session.room.localParticipant.publishData(Buffer.from(JSON.stringify({ partial: true, text })), { topic: 'lk.llm.partial' });
        } catch (e) {
          console.warn('publishData failed', e);
        }
      }

      yield chunk; // continue normal pipeline
    }
  }
}

export default defineAgent({
  entry: async (ctx: JobContext) => {
    const agent = new StreamingAgent({ instructions: 'You are helpful.' });

    const session = new voice.AgentSession({
      llm: /* your llm */,
      tts: /* optional tts */,
      stt: /* optional stt */,
    });

    await session.start({ room: ctx.room, agent });
    await ctx.connect();
  },
});
If override still doesn't run, please check and share: • Exact file where you define the custom agent and how you import/use it (so I can confirm you're passing the instance to session.start). • Any console logs/errors when agent starts. Add a log at top of 
llm_node
to verify it is called. • The LLM provider you configured (some providers may not stream; 
llm_node
won't be invoked as a stream if upstream yields only final text). If you send those details I’ll point out the exact fix. Which log or snippet would you like to share?
v
what about intercept realtime node?
t
If by "realtime node" you mean intercepting data in realtime-model agents (e.g., Gemini Live, OpenAI Realtime), override the realtime-specific pipeline hooks. Key nodes to override in Node.js: • 
realtime_audio_output_node
— intercept/modify audio frames emitted by the realtime model before they're sent to the room. • 
transcription_node
— intercept streaming transcription text from the realtime model. • 
on_user_turn_completed
— run logic after user turn ends but before the model responds. Minimal Node.js examples (adapt names for your SDK version): 1) realtime audio output (intercept frames)
Copy code
class MyAgent extends voice.Agent {
  async realtime_audio_output_node(audioStream, modelSettings) {
    const upstream = await voice.Agent.default.realtime_audio_output_node.call(this, audioStream, modelSettings);
    if (!upstream) return null;

    async function* gen() {
      for await (const frame of upstream) {
        // modify frame if needed
        yield frame;
      }
    }
    return gen.call(this);
  }
}
2) transcription node (intercept text deltas)
Copy code
class MyAgent extends voice.Agent {
  async *transcription_node(textStream, modelSettings) {
    for await (const delta of await voice.Agent.default.transcription_node.call(this, textStream, modelSettings)) {
      // inspect/modify delta
      // publish early if desired:
      await this.session.room.localParticipant.publishData(Buffer.from(JSON.stringify({partial:true, text:delta})), { topic: 'lk.llm.partial' });
      yield delta;
    }
  }
}
3) on_user_turn_completed (act before response generation)
Copy code
class MyAgent extends voice.Agent {
  async on_user_turn_completed(turnCtx, newMessage) {
    // add context, run RAG, or forward message to clients
    // then call default behavior if needed
    await voice.Agent.default.on_user_turn_completed.call(this, turnCtx, newMessage);
  }
}
Notes: • Method names/casing vary by SDK version — in v1.0.21 use the snake_case names shown in earlier examples (
realtime_audio_output_node
, 
transcription_node
, etc.). Add logging inside your overrides to confirm they run. • Realtime models may use different internal flows than the STT→LLM→TTS pipeline; choose the appropriate hook. • Publish early output using 
localParticipant.publishData(...)
on a custom topic (e.g., 
lk.llm.partial
) and have clients listen for that topic. • Ensure you yield upstream chunks/frames so the normal pipeline still runs (so ConversationItemAdded and TTS still occur). If you tell me which realtime model/provider you're using and share the agent class start code, I’ll give a concrete, version-matching snippet you can drop in. Sources: Pipeline nodes and hooks | LiveKit docs
v
yes, am using realtime llm model, are those intercepts run before onConversationItemAdded ?
t
Yes — when using a realtime LLM model, the realtime pipeline hooks run before the message is committed and the 
conversation_item_added
event fires. Order (simplified) 1. Realtime model produces output (audio/text) 2. 
realtime_audio_output_node
runs (intercept/modify audio) 3. 
transcription_node
runs (intercept/modify/forward text) 4. The processed output is committed and 
conversation_item_added
is emitted Implication: you can forward or modify tokens/frames in real time (e.g., publish partial text to a custom topic) inside those nodes, and the final, modified content will be what appears in 
conversation_item_added
. Would you like a small Node.js snippet that publishes partial tokens from 
transcription_node
(v1.0.21-compatible)? Sources: Pipeline nodes and hooks | LiveKit docs | Events and error handling | LiveKit docs | How to capture real-time agent transcriptions during conversations
v
yes, I want to intercept to get llm response before conversation_item_added, send me nodejs ts code
t
Below is a minimal Node.js (TypeScript) example for v1.0.21 that intercepts streaming LLM/transcription output before 
conversation_item_added
by overriding 
transcription_node
. It publishes partial text to a custom data topic 
lk.llm.partial
so clients receive tokens in real time. Agent file (types + imports depend on your project)
Copy code
import { defineAgent, voice, llm, type JobContext } from '@livekit/agents';

// Custom agent that intercepts transcription stream
class PartialForwardAgent extends voice.Agent {
  // v1.0.21 uses snake_case node names
  async *transcription_node(
    textStream: AsyncIterable<string>,
    modelSettings: voice.ModelSettings,
  ): AsyncIterable<string> {
    for await (const delta of textStream) {
      // delta is a partial text chunk from the realtime model
      const partial = String(delta);

      // publish partial to custom topic so clients get it early
      try {
        const lp = this.session?.room?.localParticipant;
        if (lp) {
          // send small JSON payload so client can distinguish partial/final
          const payload = JSON.stringify({ partial: true, text: partial });
          await lp.publishData(Buffer.from(payload), { topic: 'lk.llm.partial' });
        }
      } catch (err) {
        console.warn('failed to publish partial LLM chunk', err);
      }

      // forward unchanged to the rest of the pipeline
      yield delta;
    }
  }
}

export default defineAgent({
  entry: async (ctx: JobContext) => {
    const agent = new PartialForwardAgent({
      instructions: 'You are a helpful assistant.',
    });

    const session = new voice.AgentSession({
      // configure your realtime model provider here
      // e.g., llm: new RealtimeLLMProvider(...)
    });

    await session.start({
      room: ctx.room,
      agent,
      inputOptions: { audioEnabled: true, textEnabled: true },
    });

    await ctx.connect();
  },
});
Client-side reception (JS/TS)
Copy code
// listen for data_received (LiveKit JS SDK)
room.on('data_received', (payload, participant) => {
  if (payload.topic === 'lk.llm.partial') {
    const msg = JSON.parse(new TextDecoder().decode(payload.data));
    // msg.partial === true, msg.text contains partial token(s)
    console.log('LLM partial:', msg.text);
    // append to UI incrementally
  }
});
Notes and best practices • Add logging at the top of 
transcription_node
to confirm the override runs. • Ensure your realtime model actually streams partial text to trigger 
transcription_node
. • Keep publishing payloads small and consider throttling or coalescing frequent chunks. • Yield each chunk so the pipeline still commits the final item and triggers 
conversation_item_added
. • Use a custom topic name to avoid interfering with built-in streams. • If you need structured messages (final vs partial), publish { partial: true/false, text, seq } and have the client merge updates. If you want, I can provide a variant that intercepts the LLM node directly (streaming ChatChunk shape) or a version that coalesces tokens into 100ms batches before publishing. Which would you prefer?
v
can call rpc to client instead of publishData ?
t
Yes — the agent can call an RPC on the client instead of publishing data. RPCs require the client to register an RPC method and the agent to know the client's participant identity. Below are a minimal Node.js agent snippet and matching client snippet, plus pros/cons. Agent (Node.js / TypeScript) — call client RPC
Copy code
// inside your agent class or entry where `this.session`/`this.session.room` is available
async function sendPartialToClient(participantIdentity: string, text: string) {
  try {
    const lp = this.session.room.localParticipant;
    // performRpc(destinationIdentity, method, payload)
    await lp.performRpc({
      destinationIdentity: participantIdentity,
      method: 'onPartialLLM',
      payload: JSON.stringify({ partial: true, text }),
    });
  } catch (err) {
    console.warn('RPC to client failed', err);
  }
}
Call this from your `transcription_node`/`llm_node` when you want to send partial tokens. Client (JS/TS) — register RPC handler
Copy code
// run on client after joining room
room.localParticipant.registerRpcMethod('onPartialLLM', async (payloadStr) => {
  const msg = JSON.parse(payloadStr);
  // msg.partial, msg.text
  // update UI
  return 'ok';
});
How the agent finds the client identity • You must pass the target participant identity (client identity) to the agent. Common options: ◦ Include client identity in dispatch metadata (ctx.job.metadata) and read 
ctx.job.metadata
in entry. ◦ Agent can discover participants via 
session.room.participants
and pick the intended one by role/metadata. ◦ Have client send its identity to the agent via an initial publishData or RPC to agent. Pros vs publishData • RPC ◦ Pros: guaranteed delivery semantics of RPC, typed request/response, simpler request/ack flow. ◦ Cons: requires knowing destination identity, one-to-one (not broadcast), may be slower if RPC handler does heavy work. • publishData (data channel) ◦ Pros: easy broadcast to all clients, no need to know identity, good for streaming small frequent updates. ◦ Cons: no built-in RPC ack, clients must parse topic/payload. Best practices • For high-frequency partial tokens, prefer publishData (less per-message signaling). If you need ack or bidirectional fine-grained control, use RPC. • Include sequence IDs or partial/final flags so client can merge updates. • Throttle/coalesce tokens to avoid flooding RPC handlers. • Ensure client registers RPC handler before agent calls RPC. If you want, I can: • Provide a full TypeScript agent example that discovers a target participant identity and calls RPC from inside 
transcription_node
. • Provide a publishData variant that batches tokens every 100ms. Which would you prefer?
v
whatever the simple prompt is, why is minimum time to receive response from llm through agent always 2 seconds ?
t
Common causes for a ~2s minimum response time when the agent relays an LLM reply: • End‑of‑utterance/turn detection: the agent waits for endpointing (defaults include a minimum endpoint delay ~0.5s and conservative detection) before sending audio or starting LLM work. This alone often adds ~500–1500ms depending on settings. • LLM Time‑To‑First‑Token (TTFT): many models take several hundred ms to produce the first token; larger or remote models can push this to 500–1000ms+. • Network round trips & service proximity: RPCs to LLM/STT/TTS providers add latency. If services are remote from your agent host, that adds >100–500ms. • Pipeline overhead: STT → LLM → (optional) TTS processing, context assembly, and any pre/postprocessing add extra time. Each step stacks. • Default non‑streaming or buffering: if your LLM/STT/TTS provider or your agent config buffers until a chunk completes instead of streaming tokens immediately, that increases latency. • Resource contention or cold starts: model warmup, container cold starts, or overloaded instances increase latency. Ways to reduce it (practical steps) • Lower endpointing delays: tune turn detection (min/max endpointing delay, VAD/stt endpoint settings) to be more aggressive. • Use a streaming realtime model or an LLM with low TTFT (and enable streaming responses). • Co‑locate agent and model providers or choose lower‑latency endpoints/regions. • Enable preemptive/early generation features if supported (starts generation before full utterance if safe). • Avoid unnecessary pipeline steps for quick replies (disable TTS if not needed, or skip heavy postprocessing). • Reuse warm model instances (prewarm), keep model sessions alive, or use smaller/faster models for short prompts. • Measure per‑stage latency (STT, LLM TTFT, TTS) with observability to find the dominant contributor. If you share: your LLM provider/model, whether you use realtime/streaming mode, and whether STT/TTS are enabled, I’ll point to the highest‑impact changes for your setup. Sources: Turn detection and interruptions | LiveKit Docs | Agents (Python) | livekit/livekit_composite | DeepWiki | examples/voice_agents | livekit/livekit_composite | DeepWiki | examples/bank-ivr | livekit/livekit_composite | DeepWiki | Introducing LiveKit Inference: A unified model interface for voice AI
v
when using realtime ll model, when will agent open connection to llm socket ?
t
The agent opens the realtime LLM socket when the AgentSession starts — specifically during/after you call and await 
session.start()
(once the agent is connected to the LiveKit room). The connection is established lazily so the model socket is only opened while an active session is running. Practical notes: • No LLM socket is opened at agent code import or before 
session.start()
. • If you want the socket ready earlier, call 
session.start()
sooner (after prewarm) so the realtime connection is established before user audio arrives. • The socket may also be re-established on demand if the session is restarted or the model connection drops; ensure your session lifecycle handles reconnects or prewarming as needed. If you share your entry/start sequence I can point to the exact spot to add logging or prewarm calls to confirm the socket opens when you expect. Sources: Agents (Node.js) | livekit/livekit_composite | DeepWiki | OpenAI Realtime API plugin guide | LiveKit docs | Realtime model plugins | LiveKit Docs
v
when will agent start/stop stream audio to llm realtime ?
t
The agent streams audio to a realtime LLM only while an active AgentSession is running and while a user turn is active. Concretely: • When streaming starts ◦ After you call and await 
session.start()
the realtime LLM socket is established. ◦ Audio begins flowing to the model once the session is active and the agent detects user speech (via VAD or the model’s built‑in turn detection) or when you explicitly enable/send audio (e.g., 
session.input.setAudioEnabled(true)
/ RPC). ◦ For realtime models that accept raw audio, the agent sends microphone frames as soon as speech is detected (no waiting for full STT→LLM cycle). • When streaming stops ◦ Streaming stops when the user turn ends (silence detected, model endpointing, or you call a manual end such as 
session.input.setAudioEnabled(false)
/ 
commit_user_turn()
). ◦ Streaming also stops if the session is closed (
session.close()
/ 
ctx.shutdown()
), or if the realtime model socket disconnects. ◦ If you unpublish/mute the client track, the agent will stop receiving audio for that participant and thus stop sending it to the model. • Important notes ◦ Turn detection settings (VAD thresholds, min/max endpoint delays) control how quickly streaming starts/stops—tuning them reduces or increases latency. ◦ Some realtime model providers offer server‑side endpointing; in that case the model may stop the stream based on its own detection signal. ◦ If you need immediate streaming on press (push‑to‑talk), enable audio input and ensure the local track is published/unmuted and the agent’s input is enabled at press time. If you share your current turn detection/VAD settings and whether you rely on model endpointing or local VAD, I can suggest exact settings to minimize delay and ensure streaming starts/stops at the times you expect. Sources: Gemini Live API plugin | LiveKit docs | OpenAI Realtime API plugin guide | LiveKit docs | Text and transcriptions | LiveKit docs | Turn detection and interruptions | LiveKit Docs | Realtime model plugins | LiveKit Docs
v
this is the log from agent: 181906.524] INFO (85258): Creating speech handle speech_id: "speech_2cc2112e-7c6" what does "Creating speech handle" do ?
t
"Creating speech handle" means the agent is allocating a SpeechHandle object to represent a single synthesized/playout utterance. Concretely: • It’s created when the agent begins producing speech (LLM reply → TTS, or an explicit 
session.say()
/ 
generateReply()
). • The SpeechHandle tracks that speech instance (identified by the logged 
speech_id
), including state like queued, playing, interrupted, or finished. • It lets the agent: ◦ await completion (e.g., 
handle.waitForPlayout()
), ◦ detect interruptions (
handle.interrupted
), ◦ correlate logs/metrics/events for that utterance (speech_created → TTS → playout). • You’ll typically see follow-up events for the same 
speech_id
(synthesis start/complete, playout events, interrupted, finished). If you want, share the surrounding logs (speech_created, playout, interrupted) and I’ll help trace whether that handle completed or was interrupted and why. Sources: Agent speech and audio | LiveKit docs | Events and error handling | LiveKit docs
v
but I disabled the audioOutput already by setting to False, why did speech handle still create ? await session.start({ agent: this.agent, room: this.room, inputOptions: { audioEnabled: true, textEnabled: false, }, outputOptions: { audioEnabled: false, transcriptionEnabled: true, }, });
t
Good question — disabling audio output prevents the agent from publishing audio to the room, but it does not prevent the agent from creating a SpeechHandle. Why the SpeechHandle is created: • The agent still runs the internal speech pipeline (LLM → TTS) to produce text/TTS-aligned timing or to drive transcription forwarding, and the SpeechHandle represents that generated utterance lifecycle (queued, playing, interrupted, finished). • Even when 
outputOptions.audioEnabled = false
, the agent may synthesize speech internally to produce synchronized transcriptions (
lk.transcription
) or timing metadata — the SpeechHandle coordinates those steps even though no audio track is published. If you want to avoid SpeechHandle creation entirely (pure text-only behavior): • Disable any TTS or audio output features in the session/provider configuration (remove TTS provider or set TTS off). • Ensure agent code does not call 
say()
or invoke TTS nodes; rely solely on text output nodes and transcription forwarding. • Confirm 
outputOptions.textEnabled = true
and 
audioEnabled = false
, and that no pipeline nodes invoke 
tts_node
. If you share your agent code around reply generation (calls to 
generateReply()
, 
say()
, or custom nodes), I can point out the exact line causing SpeechHandle creation and suggest a code change to stop it. Sources: How to Set Up Speech-to-Text Input with Text-Only Output in LiveKit Agents | Text and transcriptions | LiveKit docs | Agent speech and audio | LiveKit docs | Events and error handling | LiveKit docs
v
I didn't call generateReply or say in agent code, how to config to disable SpeechHandle completely ?
t
You're right to expect no SpeechHandle if you never want any speech work. SpeechHandles are created whenever the agent runs the TTS/speech pipeline (even internally for transcription timing). To prevent them entirely, stop the agent from invoking any TTS work. Two ways to do that: 1) Configuration-only (preferred) • Do not configure a TTS provider in your AgentSession (remove 
tts
from the session constructor). • Keep outputOptions.audioEnabled = false and text/transcription enabled as needed. Effect: no TTS provider → agent won't run the TTS pipeline → no SpeechHandle. Example:
Copy code
const session = new voice.AgentSession({
  llm: myRealtimeLLM,   // keep LLM if needed
  // DO NOT set `tts`
  stt: mySttProvider,   // optional
});

await session.start({
  room: ctx.room,
  agent,
  inputOptions: { audioEnabled: true, textEnabled: false },
  outputOptions: { audioEnabled: false, transcriptionEnabled: true },
});
2) Code override (if you must keep a TTS provider or prevent SpeechHandle from a specific pipeline) • Override the 
tts_node
to be a no-op so the pipeline bypasses TTS and never creates a SpeechHandle. Minimal Node.js override (v1.x snake_case name):
Copy code
class NoTtsAgent extends voice.Agent {
  async *tts_node(textStream, modelSettings) {
    // consume text but do not synthesize audio or create a SpeechHandle
    for await (const chunk of textStream) {
      // optionally forward chunk downstream as text-only
      yield* []; // no audio frames produced
    }
  }
}
Which approach to use • If you never want audio/TTS at all: remove the TTS provider (approach 1). • If you sometimes need TTS but want to disable it for a particular agent or flow: override 
tts_node
to bypass synthesis (approach 2). If you share: • the AgentSession construction code (how you set 
tts
), I’ll point exactly what to remove or provide the proper 
tts_node
override tailored to your agent class.
v
I'm using realtime llm model
t
You're using a realtime LLM model, so SpeechHandles can still be created by the agent even when 
audioOutput
is disabled because the agent manages an internal speech lifecycle for the model's audio/text output. To stop SpeechHandle creation entirely, do one of the following: 1) Preferred — disable audio modality (configuration) • Start the session so the agent uses text-only modality for the realtime model (no audio output). Remove or set the model/session to text-only so the agent does not run the speech lifecycle. ◦ Example: configure the realtime model with modalities set to ["text"] or do not enable any audio output providers/voices. ◦ Ensure you do NOT provide a TTS/audio provider in the session config. • Keep outputOptions.audioEnabled = false and text/transcription enabled as needed. 2) If you must keep realtime model but suppress speech handles for specific flows — override the realtime audio/tts node • Implement a no-op 
realtime_audio_output_node
/ 
tts_node
(snake_case names in v1.x) that consumes model audio/text but produces no SpeechHandle/audio frames. • Example pattern (Node.js, v1.x):
Copy code
class NoSpeechAgent extends voice.Agent {
    async *realtime_audio_output_node(audioStream, modelSettings) {
      // consume upstream but do not yield audio frames -> no playout/speech handle
      for await (const _frame of await voice.Agent.default.realtime_audio_output_node.call(this, audioStream, modelSettings)) {
        // drop frames
      }
      // end
    }
  }
• Or override 
tts_node
similarly to avoid creating TTS playback handles. Notes & caveats • Realtime models often produce audio internally; switching the model to text-only or removing TTS is the cleanest way to avoid speech lifecycle artifacts. • If you still want model audio for other uses, but want to avoid SpeechHandles for certain responses, use the node override approach selectively. • After these changes, verify by checking logs for speech-related ids/events; SpeechHandle creation logs should no longer appear. If you share how you construct the realtime model in your session (the exact AgentSession/RealtimeModel config), I’ll give the exact code change to disable modalities or the exact node override you can drop into your agent class. Which would you prefer — config change (modalities/text-only) or node override? Sources: Realtime model plugins | LiveKit Docs | OpenAI and LiveKit partner to turn Advanced Voice into an API | How to Set Up Speech-to-Text Input with Text-Only Output in LiveKit Agents | Simple Content Filter | LiveKit docs | Pipeline nodes and hooks | LiveKit docs
v
give me implement of realtime_audio_output_node
t
Below are two TypeScript examples you can drop into your agent class (v1.0.21 style). Both override 
realtime_audio_output_node
(snake_case) and show how to consume the upstream audio stream, optionally modify frames, and yield frames downstream. Notes: • Use the snake_case node name 
realtime_audio_output_node
for v1.x. • Ensure your agent instance is passed to 
session.start(...)
. • Audio frame type depends on your SDK; treat frames as opaque objects and modify fields you know (e.g., samples, volume). • Always yield upstream frames (or modified frames) so normal pipeline continues; to suppress playback, simply do not yield frames. 1) Pass-through (log & forward frames)
Copy code
import { voice, llm } from '@livekit/agents';

class MyAgent extends voice.Agent {
  // keep same signature as SDK expects
  async *realtime_audio_output_node(audioStream: AsyncIterable<any>, modelSettings: voice.ModelSettings): AsyncIterable<any> {
    // get upstream generator (if default exists) and use it, else use provided audioStream
    const upstream = await (voice.Agent as any).default.realtime_audio_output_node?.call(this, audioStream, modelSettings) ?? audioStream;

    for await (const frame of upstream) {
      try {
        // Inspect or log minimal info (avoid heavy work here)
        // console.log('audio frame timestamp', frame?.timestamp);

        // Optional: modify frame (example: adjust a hypothetical volume field)
        // if (frame && typeof frame.volume === 'number') frame.volume *= 0.9;

      } catch (err) {
        console.warn('realtime_audio_output_node processing error', err);
      }

      // yield the (possibly modified) frame downstream so playback occurs
      yield frame;
    }
  }
}
2) No-op / suppress playback (consume but do not yield)
Copy code
import { voice } from '@livekit/agents';

class SilentAgent extends voice.Agent {
  async *realtime_audio_output_node(audioStream: AsyncIterable<any>, modelSettings: voice.ModelSettings): AsyncIterable<any> {
    const upstream = await (voice.Agent as any).default.realtime_audio_output_node?.call(this, audioStream, modelSettings) ?? audioStream;

    for await (const frame of upstream) {
      // consume frames to keep pipeline moving, but do NOT yield them.
      // This prevents audio playout and avoids creating playout SpeechHandle.
      // Optionally perform light logging or forward partial text via publishData/RPC.
    }

    // generator ends; no frames yielded => no playback
  }
}
Usage (in your agent entry)
Copy code
// create agent instance
const agent = new MyAgent({ instructions: '...' });

// create session with realtime model config
const session = new voice.AgentSession({
  llm: /* your realtime model provider */,
  // no explicit tts provider if you want text-only behavior
});

await session.start({ room: ctx.room, agent, inputOptions: { audioEnabled: true }, outputOptions: { audioEnabled: false } });
await ctx.connect();
Best practices • Keep per-frame work minimal; avoid heavy synchronous work in the loop. • If modifying audio samples, be certain of the frame structure (samples, channels, sample rate) for your provider. • If you suppress frames to prevent SpeechHandle creation, also verify transcription/timing requirements — you may need to forward text via 
transcription_node
or publish custom data messages so clients still receive responses. • Add small, infrequent logging only for debugging; too much logging in the audio loop can affect performance. If you want, tell me: • the realtime LLM provider you use (OpenAI/Gemini), and whether you need a safe example that scales audio frames (actual sample buffer manipulation), and I’ll provide a variant that shows sample-level volume scaling.
v
added override async realtimeAudioOutputNode but it doesn't trigger: override async realtimeAudioOutputNode( audioStream*:* ReadableStream<AudioFrame>, _modelSettings*:* voice.ModelSettings )*:* Promise<ReadableStream<AudioFrame> | null> { // Consume and drop audio so no TTS/audio is produced utils.log('>>> realtimeAudioOutputNode triggered'); return *null*; }
t
Because you set audio output off, the agent skips the audio pipeline entirely — so your 
realtimeAudioOutputNode
will never be invoked. To intercept realtime model audio you must enable audio output, consume the stream in your node, and return null (or yield modified frames) to control whether audio is published. Minimal working pattern (TypeScript, v1.x): 1) Start session with audio enabled:
Copy code
await session.start({
  room: ctx.room,
  agent,
  outputOptions: { audioEnabled: true, transcriptionEnabled: true },
  inputOptions: { audioEnabled: true, textEnabled: false },
});
2) Override and consume the audio stream (prevents playback by returning null):
Copy code
class MyAgent extends voice.Agent {
  async realtime_audio_output_node(audioStream: AsyncIterable<any>, _modelSettings: any): Promise<AsyncIterable<any> | null> {
    console.log('>>> realtime_audio_output_node triggered');

    // consume upstream to avoid backpressure, but do not yield frames -> suppress playout
    (async () => {
      try {
        for await (const frame of audioStream) {
          // lightweight inspection/logging only
          // e.g., console.debug('frame ts', frame?.timestamp);
        }
      } catch (err) {
        console.warn('consume audioStream error', err);
      }
    })();

    // returning null prevents audio from being published to the room
    return null;
  }
}
Notes • Keep per-frame work minimal — heavy work will hurt realtime performance. • If you only need text output, enable 
audioEnabled: true
temporarily so the node runs, consume frames, and rely on 
lk.transcription
for text; or configure your realtime model/modality to text-only to avoid audio generation altogether. • Add logging inside the node to verify it runs. If it still doesn't trigger after enabling audio, share your session.start config and agent instantiation and I’ll inspect. Sources: Text and transcriptions | LiveKit docs | Pipeline nodes and hooks | LiveKit docs
v
triggered now after enabled, but it triggered after "Creating speech handle" event
t
That's expected: the SpeechHandle represents the logical response lifecycle and is created when the agent commits to producing speech (LLM → response), before the audio frames are processed by the realtime audio node. Key points: • SpeechHandle creation happens once the agent begins generating a reply; it is created prior to audio-frame processing and before 
realtime_audio_output_node
runs. • Returning 
null
from 
realtime_audio_output_node
prevents audio from being published (suppressing playout) but does not stop the SpeechHandle from being created. • If you want no SpeechHandle at all, you must prevent the agent from producing audio at the model/pipeline level (use a text-only modality or remove/disable TTS/realtime audio output in the session/model config). Overriding 
realtime_audio_output_node
can only control frames, not the handle creation itself. If you want, tell me whether you prefer: • suppressing playback but keeping the logical speech lifecycle (current approach), or • preventing SpeechHandle creation entirely (I’ll show how to set the realtime model/session to text-only or remove TTS).
v
preventing SpeechHandle creation entirely
t
To prevent SpeechHandle creation entirely when using a realtime LLM, you must stop the agent from producing any audio at the model level (not just suppressing playout). Two recommended approaches — pick one: 1) Configure the realtime model to text-only (preferred) • Set the realtime model to use text modality only so it never generates audio. • Do not provide a TTS/voice configuration. Node.js example (pseudo-code; adapt to your provider):
Copy code
const session = new voice.AgentSession({
  llm: new RealtimeModelProvider({
    model: 'gpt-realtime',
    modalities: ['text'], // critical: request text-only modality
    // no `voice` / no audio config
  }),
  // no tts provider
});

await session.start({
  agent,
  room: ctx.room,
  inputOptions: { audioEnabled: true },          // accept user speech if needed
  outputOptions: { audioEnabled: false, transcriptionEnabled: true }, // no audio out
});
Effect: the model will not synthesize audio, so no SpeechHandle is created. 2) If you cannot change model modalities, override realtime audio node to bypass audio generation at upstream entry • Ensure audio output is enabled so the node is invoked, but prevent speech lifecycle by intercepting the model’s audio before SpeechHandle is created is NOT reliable — SpeechHandle is created earlier for realtime models. Instead, configure the model to not generate audio or remove the TTS pathway. If you share which realtime provider and how you construct the RealtimeModel (exact code), I’ll give the precise config for that provider (OpenAI Realtime / Gemini Live / etc.) to ensure text-only operation. Which provider are you using? Sources: Text and transcriptions | LiveKit docs | Realtime model plugins | LiveKit Docs | Agent speech and audio | LiveKit docs | Events and error handling | LiveKit docs
v
using "gpt-realtime-mini". SpeechHandle still created with this config!: const session = new voice.AgentSession({ llm: new ManualRealtimeModel({ model: 'gpt-realtime-mini', turnDetection: null , modalities: ['text']}), turnDetection: 'manual', voiceOptions: { preemptiveGeneration: false, }, }); class ManualRealtimeModel extends openai.realtime.RealtimeModel { override get capabilities() { return { ...super.capabilities, turnDetection: false, modalities: ['text']}; } } await session.start({ agent: this.agent, room: this.room, inputOptions: { audioEnabled: true, textEnabled: false, }, outputOptions: { audioEnabled: false, transcriptionEnabled: true, }, });
t
You're using gpt‑realtime‑mini and still seeing SpeechHandle creation. That is expected behaviour in the current SDK: SpeechHandles represent the agent's response lifecycle (generation/planning) and can be created even when no audio is published. They are not an indication that audio will be played. What you can do now (choose one): 1) Keep text‑only behaviour and safely ignore SpeechHandles (recommended) • Leave your session as-is (modalities: ['text'], outputOptions.audioEnabled = false). • Consume responses via the conversation item event or transcription: ◦ Listen to ConversationItemAdded to get the final text. ◦ Or register the 
lk.transcription
text stream for realtime text. • If you need to observe the handle for logging, just ignore awaiting it — it won't cause audio playback. 2) If you must avoid SpeechHandle side-effects entirely, use a provider/config that guarantees no speech lifecycle • Ensure the realtime model is truly text‑only at the provider level (modalities=['text'] and audioOutput=false in the model/session provider). If the provider still signals "speech" lifecycle, the SDK may still emit speech events. • If your provider supports a strict text-only realtime mode, use that exact provider flag (I can craft the exact config if you tell me the provider / plugin class you instantiate). 3) Workaround: intercept and no‑op the speech hooks • You cannot reliably stop creation, but you can neutralize effects by hooking speech events and making them no-ops: ◦ session.on('speech_created', ev => {/* noop or immediate resolve */}) ◦ override 
realtime_audio_output_node
or 
tts_node
to consume but not yield audio (you already did this). If you want a concrete change now, tell me: • Which exact provider/class you instantiate for the realtime model (the code where you do 
new ManualRealtimeModel(...)
/ 
new openai.realtime.RealtimeModel(...)
), and • Whether you prefer: (A) a provider config that enforces strict text-only, or (B) a small snippet to neutralize speech events (session.on('speech_created') handler + 
realtime_audio_output_node
no-op). I’ll give the exact code for your choice. Sources: OpenAI Realtime API plugin guide | LiveKit docs | Azure OpenAI Realtime API and LiveKit | LiveKit docs | Realtime model plugins | LiveKit Docs | Events and error handling | LiveKit docs
v
how to disable text output, audio output of realtime llm model since we ask llm to use tool call to submit result to client already ?
t
You can suppress both text and audio outputs for a realtime model so the agent only uses tool calls. Two reliable approaches — choose one based on your provider and needs. 1) Preferred — configure the realtime model/session for no outputs • Ensure the realtime model is requested as text-only (modalities/text-only) and disable session outputs: ◦ Node.js start options:
Copy code
await session.start({
      agent,
      room: ctx.room,
      inputOptions: { audioEnabled: true, textEnabled: false }, // accept audio input if needed
      outputOptions: { audioEnabled: false, transcriptionEnabled: false }, // disable both outputs
    });
• Ensure your RealtimeModel is configured to not produce audio/text automatically (provider-specific flag). Example pseudo:
Copy code
new RealtimeModel({ model: 'gpt-realtime-mini', modalities: ['text'], audioOutput: false })
• Make agent instructions explicit: “Output results only via tool calls; do not emit text or speech.” 2) If the provider or SDK still creates speech artifacts, neutralize outputs in nodes/events • Override audio/text nodes to drop output (no playout/transcription forwarding). ◦ Suppress audio:
Copy code
async realtime_audio_output_node(audioStream, modelSettings) {
      // consume upstream to avoid backpressure, but do NOT yield frames
      (async () => { for await (const _ of audioStream) {} })();
      return null;
    }
• Suppress transcription forwarding (publish nothing):
Copy code
async *transcription_node(textStream, modelSettings) {
      // consume but don't yield/publish transcription
      for await (const _ of textStream) {}
    }
• Hook or ignore speech events so SpeechHandle effects are inert:
Copy code
session.on('speech_created', ev => { /* no-op */ });
Notes and cautions • Some realtime providers may still trigger internal speech lifecycle events (SpeechHandle) even when outputs are disabled; that’s harmless if you suppress node outputs and ignore events. • Make agent instructions enforce tool-only output to avoid the LLM generating freeform text. • If you need absolute assurance the provider won’t produce audio/text, use a provider flag that enforces text-only modality (provider-specific). Tell me which provider/plugin class you use (OpenAI Realtime, Gemini Live, etc.) and I’ll give the exact config snippet. Do you want the exact Node.js code for your current provider (you mentioned gpt-realtime-mini)? Sources: Text and transcriptions | LiveKit docs | Realtime model plugins | LiveKit Docs | OpenAI Realtime API plugin guide | LiveKit docs | Gemini Live API plugin | LiveKit docs | Pipeline nodes and hooks | LiveKit docs
v
OpenAI Realtime gpt-realtime-mini
t
To use OpenAI Realtime (gpt-realtime-mini) and have the agent only emit tool calls (no audio, no transcription/text), do two things: 1) configure the Realtime model for text-only/tool-call behavior, and 2) disable session output plus optionally neutralize pipeline nodes/events. Below is a focused Node.js example and the exact steps. 1) Provider + model config (text-only / no audio) • Ensure the Realtime model is requested with text-only modality and no voice/audio config so the provider will not generate audio. • Example (pseudo / adapt to your plugin constructor):
Copy code
import * as openai from '@livekit/agents-plugin-openai';
const realtimeModel = new openai.realtime.RealtimeModel({
  model: 'gpt-realtime-mini',
  modalities: ['text'],      // request text-only
  audioOutput: false,        // explicit if supported by provider wrapper
  turnDetection: false,      // you'll handle turns manually or via tools
});
2) Start session with outputs disabled • Disable both audio and transcription forwarding in session.start:
Copy code
const session = new voice.AgentSession({ llm: realtimeModel /* no tts */ });

await session.start({
  agent,
  room: ctx.room,
  inputOptions:  { audioEnabled: true,  textEnabled: false }, // accept audio input if needed
  outputOptions: { audioEnabled: false, transcriptionEnabled: false }, // disable both outputs
});
3) Enforce tool-only responses in prompts/instructions • Tell the model to only use tool calls and never output text/audio:
Copy code
"You are an assistant that MUST return results only via the 'submit_result' tool. Do NOT emit any text or speech. If you would normally respond, call the tool with the structured result."
• Use function/tool schema for the tool and ensure the model is given the tool definitions. 4) Optional safety: neutralize nodes & events in agent • If the provider or SDK still triggers speech events, neutralize them:
Copy code
// ignore speech lifecycle
session.on('speech_created', () => {/* noop */});

// suppress any audio node if invoked
class NoAudioAgent extends voice.Agent {
  async realtime_audio_output_node(audioStream, _settings) {
    (async () => { for await (const _ of audioStream) {} })(); // consume
    return null; // prevent publish
  }
  async *transcription_node(textStream, _settings) {
    for await (const _ of textStream) { /* drop */ }
    // do not yield => no transcription forwarded
  }
}
• Use this only if you still see speech/transcription events after steps 1–3. 5) Validate tool-only behavior • Test with a prompt that triggers a tool call; verify: ◦ No lk.transcription messages are sent ◦ No audio track is published ◦ Model calls your tool and you receive the tool payload on the agent side Notes / caveats • Some SDK/provider implementations still create logical SpeechHandles or emit speech events even when outputs are disabled; those are inert if audio/transcription are disabled and can be safely ignored or suppressed with the event handlers above. • Make the instruction to the model strict and include tool schemas; using 
modalities: ['text']
plus 
audioOutput:false
is the most reliable approach. • If you want, I can produce the exact constructor args for the OpenAI plugin class you use (show me the code line where you instantiate the RealtimeModel) and a drop-in agent example that performs only tool calls. Which plugin/class are you instantiating now?
Open in Slack
PreviousNext
Powered by