When handing off from Agent A to Agent B, I’m noticing that the time it takes for Agent B to respond to voice input is on the order of seconds (~10s). How can I mitigate this issue? It causes unpleasant interruptions in conversation flow.

Thanks for reaching out, Gabriel. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Explain the first two bullets

You asked for more detail on the first two mitigation strategies: keeping agents warm (disabling cold starts) and prewarming heavy dependencies. Below are concise explanations of why each reduces handoff latency and simple examples of how they work. 1) Keep agents warm (disable cold starts) • Why it helps: Cold starts add time because the agent process must be created, networking established, and core components initialized before it can accept audio. If an agent is already running (warm), those steps are already done, so it can start processing audio immediately when handed off. This removes the multi-second delay caused by startup and room-join overhead. • When to use it: Use on paid/managed plans or any deployment where you can afford to keep at least one agent instance running continuously. • Practical effect: Response time moves from many seconds (cold start) down to sub-second or low-single-second latency because only live connection and processing happen at handoff, not full initialization. 2) Prewarm heavy dependencies (load models once, reuse) • Why it helps: Large models and libraries (VAD, STT, LLM, TTS) take time to load into memory and initialize. If each agent session loads them on-demand, that load time repeats and adds to handoff latency. Prewarming loads these resources ahead of any session so the agent can reuse them instantly. • How to implement (pattern): ◦ During agent startup or a prewarm hook, load heavy models and store them on the process-level context (for example,

proc.userdata

or a global cache). ◦ When a new session/job starts, read the preloaded models from that shared store instead of reloading. • Small pseudo-example (conceptual): ◦ On server start:

proc.userdata["vad"] = load_vad_model()

◦ In session handler:

vad = ctx.proc.userdata["vad"]

and use it immediately to process incoming audio • Practical effect: Eliminates repetitive model-load time per handoff; combined with a warm agent, this can reduce handoff latency from ~10s to well under a couple seconds. If you tell me which runtime or framework your agents use (Python, Node, etc.), I can provide a short concrete code snippet showing a prewarm hook and reading the cached models inside a session.

Node

You're asking for Node.js examples and explanation for (1) keeping agents warm and (2) prewarming heavy dependencies. Below are concise, actionable explanations plus Node.js patterns you can adapt. 1) Keep agents warm • Why: If an agent process is already running, there’s no process startup, no room-join delay, and no model initialization at handoff — so latency drops from many seconds to sub-second or low-single-second. • How to do it (patterns): ◦ Managed/paid plan: Configure your deployment so agent instances are not scaled to zero. This is a configuration/plan choice rather than code. ◦ Self-hosted: Run at least one persistent agent process per agent type. Use a process manager (pm2, systemd, Docker with restart policy) to ensure uptime. • Node pattern (pm2 example): a. Start agent server with pm2 to keep it always running: • pm2 start agent-server.js --name my-agent a. Ensure health checks and auto-restart: • pm2 save • pm2 startup This ensures the agent process stays warm and ready for immediate handoff. 2) Prewarm heavy dependencies (Node.js) • Why: Loading VAD/STT/LLM/TTS libraries on each session causes repeated multi-second delays. Load them once on process start and reuse. • Implementation pattern: ◦ Load models during agent process startup and store on a module-level cache or

proc.userdata

equivalent. ◦ In per-session handlers, read from that cache instead of reloading. • Minimal Node.js example (conceptual):

// agent-server.js (process-level)
const vad = require('./vad'); // module that exposes async load()
const stt = require('./stt');
const tts = require('./tts');

const cache = {};

async function prewarm() {
  cache.vad = await vad.load();   // load once
  cache.stt = await stt.load();   // load once
  cache.tts = await tts.load();   // load once
  console.log('Prewarm complete');
}

prewarm().catch(err => { console.error('Prewarm failed', err); process.exit(1); });

// Export cache so session handlers can access it
module.exports = { cache };

// session-handler.js (per-hand-off/session)
const { cache } = require('./agent-server');

async function handleSession(session) {
  // Ensure we connect to the room early (start receiving audio)
  await session.connect(); // pseudo-call; call your LiveKit connect flow

  // Use preloaded models immediately
  const vadModel = cache.vad;
  const sttModel = cache.stt;
  const ttsModel = cache.tts;

  // Process audio with vadModel/sttModel...
}

• Tips: ◦ Prewarm during process start, not first session. If you must prewarm lazily, do it on a warmup endpoint that you call before handoff. ◦ Keep cache references global to avoid garbage collection. ◦ If a model exposes a streaming or session API, reuse client connections rather than creating new ones per session. If you share which Node libraries or services you use for VAD/STT/LLM/TTS (example: Silero, Whisper, OpenAI, local LLM), I’ll provide a concrete snippet using those libraries and show where to connect to the LiveKit room before loading noncritical resources. Would you like that?

It’s TypeScript, so let’s go ahead and rewrite those for TS

Below are concise TypeScript examples and brief explanations for (1) keeping agents warm and (2) prewarming heavy dependencies. 1) Keep agents warm • Why: A running process avoids process startup, room-join, and model-init delays. • How to do it: ◦ Managed: configure your plan/deployment to keep at least one agent instance running. ◦ Self-hosted: run a persistent agent process with a process manager (pm2, systemd, Docker restart policy). TypeScript example (pm2-friendly entry):

// src/agent-server.ts
import { startAgentServer } from './server'; // your server bootstrap
import { prewarm } from './prewarm';

async function main() {
  // Start server so process stays alive and ready for handoffs
  await startAgentServer();

  // Optionally prewarm heavy deps here (see next section)
  await prewarm();

  console.log('Agent server running and warm');
}

main().catch((err) => {
  console.error('Fatal startup error', err);
  process.exit(1);
});

Start with pm2 to keep it running: • pm2 start dist/agent-server.js --name my-agent • pm2 save && pm2 startup 2) Prewarm heavy dependencies • Why: Load VAD/STT/LLM/TTS once on process start and reuse to avoid repeated multi-second loads. • Pattern: load models into a module-level cache, connect to LiveKit room early in session handler, then use cached models. TypeScript prewarm + session example:

// src/cache.ts
export interface Cache {
  vad?: any;
  stt?: any;
  tts?: any;
  // add types for your libs
}

export const cache: Cache = {};

// src/prewarm.ts
import { cache } from './cache';
import { loadVad } from './vad';
import { loadStt } from './stt';
import { loadTts } from './tts';

export async function prewarm() {
  // Load heavy models once at process start
  cache.vad = await loadVad();   // e.g., Silero VAD loader
  cache.stt = await loadStt();   // e.g., Whisper or streaming STT client
  cache.tts = await loadTts();   // e.g., shared TTS client
  console.log('Prewarm complete');
}

// src/session-handler.ts
import { cache } from './cache';
import type { LiveKitSession } from './types'; // replace with your types

export async function handleSession(session: LiveKitSession) {
  // Connect early so audio starts flowing
  await session.connect(); // your LiveKit connect call

  // Use cached models immediately
  const vad = cache.vad!;
  const stt = cache.stt!;
  const tts = cache.tts!;

  // Example flow (pseudo):
  // 1. Use vad to detect speech frames
  // 2. Stream to stt for transcription
  // 3. Use LLM + tts for responses
}

Implementation tips • Prewarm at process start (not on first session). If needed, expose a warmup HTTP endpoint and call it before handoff. • Keep cache references module-scoped to avoid GC and reloading. • If libraries provide client connections (streaming STT/LLM), reuse connections rather than recreating per session. • Connect to the LiveKit room before loading nonessential models to reduce perceived delay. If you share which TypeScript libraries/services you use for VAD/STT/LLM/TTS (examples: Silero, Whisper, OpenAI, local LLM), I’ll provide a concrete snippet showing how to load and reuse those specific clients. Would you like that?

Silero VAD, OpenAI SST, Google LLM, Cartesia TTS

You're asking for TypeScript examples and concrete guidance for (1) keeping agents warm and (2) prewarming Silero VAD, OpenAI STT, Google LLM, and Cartesia TTS. Below are concise, production-ready TypeScript patterns you can drop into your agent server. 1) Keep agents warm • Goal: keep at least one agent process running so handoffs avoid cold-start initialization. • How: run a persistent agent process (managed plan or self-hosted) with a process manager. Example entry (pm2 / Docker friendly):

// src/agent-server.ts
import { startAgentServer } from './server';
import { prewarm } from './prewarm';

async function main() {
  // Start server to keep the process alive for immediate handoffs
  await startAgentServer();

  // Prewarm heavy deps (optional here or handled by process start)
  await prewarm();

  console.log('Agent server running and warm');
}

main().catch((err) => {
  console.error('Fatal startup error', err);
  process.exit(1);
});

Run with pm2: • pm2 start dist/agent-server.js --name my-agent • pm2 save && pm2 startup 2) Prewarm heavy dependencies (Silero VAD, OpenAI STT, Google LLM, Cartesia TTS) • Goal: load large models/clients once at process start and reuse them across sessions so handoffs only need routing/connect work. Cache module:

// src/cache.ts
export interface Cache {
  vad?: any;
  stt?: any;
  llm?: any;
  tts?: any;
}
export const cache: Cache = {};

Prewarm implementation:

// src/prewarm.ts
import { cache } from './cache';
import { loadSileroVAD } from './silero';
import { createOpenAIClient } from './openai';
import { createGoogleLLM } from './google';
import { createCartesiaTTS } from './cartesia';

export async function prewarm() {
  // Silero VAD: load model once
  cache.vad = await loadSileroVAD();

  // OpenAI STT: create and warm client (Whisper-style or streaming client)
  cache.stt = createOpenAIClient({ apiKey: process.env.OPENAI_API_KEY });

  // Google LLM: client/config ready (auth/credentials established)
  cache.llm = createGoogleLLM({ credentials: process.env.GOOGLE_CREDENTIALS });

  // Cartesia TTS: client instance or warmed connection
  cache.tts = createCartesiaTTS({ apiKey: process.env.CARTESIA_API_KEY });

  console.log('Prewarm complete: VAD, STT, LLM, TTS ready');
}

Session handler that connects early and reuses cache:

// src/session-handler.ts
import { cache } from './cache';
import type { LiveKitSession } from './types';

export async function handleSession(session: LiveKitSession) {
  // Connect to LiveKit room immediately so audio flow begins
  await session.connect();

  // Use prewarmed components
  const vad = cache.vad!;
  const sttClient = cache.stt!;
  const llmClient = cache.llm!;
  const ttsClient = cache.tts!;

  // Example flow (pseudo):
  // 1) Run VAD on incoming frames to detect speech
  // 2) Stream speech to sttClient for transcription
  // 3) Send transcription to llmClient for reply generation
  // 4) Synthesize reply via ttsClient and play back in the room

  // Keep handlers non-blocking; stream where possible to minimize latency
}

Implementation notes and tips • Silero VAD (Node/TS): load model artifact once (file or package), keep model object in

cache.vad

. Use VAD for quick turn detection so you send minimal audio to STT. • OpenAI STT: create a persistent OpenAI client instance and stream audio chunks to it rather than creating clients per session. Reuse HTTP/HTTP2 connections if the client supports it. • Google LLM: initialize client and auth at prewarm; reuse the client for generate calls. If you use streaming responses, reuse connections where supported. • Cartesia TTS: create a persistent TTS client or warm a session so synth calls return fast. • Connect before heavy work: always call

session.connect()

early so LiveKit audio can flow while the agent is already prepared to handle it. • Health/warm endpoints: if you cannot prewarm on startup, add a warmup HTTP endpoint that triggers

prewarm()

so an orchestrator can call it before handoff. • Keep cache references module/global-scoped to avoid GC and unnecessary reloads. If you want, I can: • Provide concrete TypeScript code for

loadSileroVAD

,

createOpenAIClient

,

createGoogleLLM

, and

createCartesiaTTS

using the exact libraries you plan to use, or • Convert these patterns into a complete LiveKit AgentServer/defineAgent example that hooks into

JobProcess.userData

if you prefer that storage surface. Which would you like? Sources: Silero VAD plugin | LiveKit docs | Google Gemini LLM plugin guide | LiveKit Docs | Cartesia TTS plugin guide | LiveKit docs

Convert the patterns into a complete

defineAgent

Below is a complete TypeScript

defineAgent

example that keeps the agent process warm and prewarms Silero VAD, OpenAI STT, Google LLM, and Cartesia TTS. It uses a module-level cache and the

prewarm

hook so heavy dependencies are loaded at process start and reused across sessions. Adjust import paths and client constructors to match your packages.

// src/agent.ts
import { defineAgent, JobProcess, JobContext, cli } from '@livekit/agents';
import type { VAD } from '@livekit/agents-plugin-silero'; // adjust to actual types
import * as silero from '@livekit/agents-plugin-silero';
import * as openai from '@livekit/agents-plugin-openai';
import * as google from '@livekit/agents-plugin-google';
import * as cartesia from '@livekit/agents-plugin-cartesia';
import { fileURLToPath } from 'node:url';

// Module-level cache shared across sessions in this process
interface Cache {
  vad?: VAD;
  stt?: any;   // OpenAI STT client type
  llm?: any;   // Google LLM client type
  tts?: any;   // Cartesia TTS client type
}
const cache: Cache = {};

// Optional: helper loaders (implement according to your libs)
async function loadSileroVAD(): Promise<VAD> {
  return await silero.VAD.load(); // adjust as needed
}
function createOpenAIClient(opts: { apiKey?: string }) {
  return new openai.STT({ apiKey: opts.apiKey, model: 'whisper-1' });
}
function createGoogleLLM(opts: { credentials?: string }) {
  return new google.LLM({ credentials: opts.credentials, model: 'gemini-1' });
}
function createCartesiaTTS(opts: { apiKey?: string }) {
  return new cartesia.TTS({ apiKey: opts.apiKey, model: 'sonic-3' });
}

// defineAgent with prewarm hook and entry handler
export default defineAgent({
  // Prewarm: run once when the agent process is started/warmed
  prewarm: async (proc: JobProcess) => {
    console.log('Prewarm starting...');

    // Silero VAD: load model into cache and JobProcess.userData for redundancy
    cache.vad = await loadSileroVAD();
    proc.userData.vad = cache.vad;

    // OpenAI STT: create client and store
    cache.stt = createOpenAIClient({ apiKey: process.env.OPENAI_API_KEY });
    proc.userData.stt = cache.stt;

    // Google LLM: create client and store
    cache.llm = createGoogleLLM({ credentials: process.env.GOOGLE_CREDENTIALS });
    proc.userData.llm = cache.llm;

    // Cartesia TTS: create client and store
    cache.tts = createCartesiaTTS({ apiKey: process.env.CARTESIA_API_KEY });
    proc.userData.tts = cache.tts;

    console.log('Prewarm complete: VAD, STT, LLM, TTS loaded');
  },

  // Entry: called per job/session; reuse prewarmed components
  entry: async (ctx: JobContext) => {
    // Prefer proc.userData (LiveKit storage) but fallback to module cache
    const vad = (ctx.proc.userData.vad ?? cache.vad) as VAD;
    const sttClient = ctx.proc.userData.stt ?? cache.stt;
    const llmClient = ctx.proc.userData.llm ?? cache.llm;
    const ttsClient = ctx.proc.userData.tts ?? cache.tts;

    if (!vad || !sttClient || !llmClient || !ttsClient) {
      throw new Error('One or more prewarmed components are missing');
    }

    // Connect early so LiveKit audio starts flowing while we prepare processing
    await ctx.connect(); // ensure your JobContext implements connect()

    // Example: set up a quick turn-detection -> stt -> llm -> tts pipeline
    const session = ctx.createVoiceSession?.() ?? null; // replace with your session construct
    if (!session) {
      throw new Error('Voice session API not available on this context');
    }

    // Wire VAD: listen to incoming audio, detect speech segments
    session.on('audio', async (audioChunk: Buffer) => {
      // 1) Quick VAD check
      const isSpeech = await vad.isSpeech(audioChunk); // adapt to actual API
      if (!isSpeech) return;

      // 2) Stream to STT (example: send chunk to OpenAI STT client)
      const transcript = await sttClient.transcribeStreamSegment(audioChunk);

      // 3) Query LLM (Google) for a reply
      const replyText = await llmClient.generate({ prompt: transcript, maxTokens: 256 });

      // 4) Synthesize with Cartesia TTS and play back
      const audioOut = await ttsClient.synthesize(replyText);
      await session.playAudioBuffer(audioOut);
    });

    // Optional: initiate a quick greeting to verify readiness
    const readyText = 'Agent ready';
    const readyAudio = await ttsClient.synthesize(readyText);
    await session.playAudioBuffer(readyAudio);

    // Keep job running until session ends; job lifecycle handled by LiveKit agent infra
    await ctx.waitForSessionEnd?.();
  },
});

// Run via CLI worker (adjust import if necessary)
if (require.main === module) {
  const entryPath = fileURLToPath(import.meta.url);
  cli.runApp({ agent: entryPath });
}

Notes and adjustments you should make • Replace placeholder APIs (e.g.,

vad.isSpeech

,

transcribeStreamSegment

,

createVoiceSession

,

playAudioBuffer

) with the actual methods from your Silero/OpenAI/Google/Cartesia client libraries or LiveKit helpers. • Ensure

ctx.connect()

,

ctx.createVoiceSession()

, and

ctx.waitForSessionEnd()

are implemented by your LiveKit JobContext; adapt the example to your actual SDK surface. • Keep credentials in environment variables and ensure network connections are warmed where supported (HTTP keep-alive, streaming sessions). • If you prefer storing prewarmed clients only in

proc.userData

, you can remove the module-level `cache`; module cache is useful for local dev and redundancy. Would you like a version that includes concrete client-call examples for Silero, OpenAI, Google, and Cartesia methods (based on the exact npm packages you use)?

The agent is transferring much faster now, thank you. How do I make them speak first once transferred?

Great — to make the transferred agent speak first, implement its onEnter lifecycle hook and trigger a TTS or generated reply as soon as the agent becomes active. Use

session.say()

for a fast scripted greeting or

session.generateReply()

/

session.generateReply({ instructions })

to produce an LLM-driven reply. Concise TypeScript example (fits into your existing

defineAgent

pattern):

class SupportAgent extends voice.Agent {
  constructor() {
    super({ instructions: 'You are a support specialist.' });
  }

  // Called when this agent becomes active after a handoff
  async onEnter(): Promise<void> {
    // Fast scripted TTS greeting (lowest latency)
    await this.session.say('Hi, I am Sarah from support. How can I help you today?');

    // OR — LLM-generated reply (more flexible, slightly higher latency)
    // await this.session.generateReply({
    //   instructions: 'Greet the caller and ask how you can assist.'
    // });
  }
}

Where to place it • Add this agent class as the target in your handoff tool so when the previous agent returns the new agent, LiveKit will activate it and run

onEnter

. • Keep the prewarmed TTS client available so

session.say()

returns immediately. Notes • Use

session.say()

when you need the fastest possible spoken greeting. • Use

session.generateReply()

when you want context-aware or dynamic replies (it will call the LLM first). • If you still see a short delay, ensure the TTS client is prewarmed and the agent process is kept warm so

onEnter

can execute immediately. Want a complete

defineAgent

snippet with this

SupportAgent

wired into a handoff example? Sources: Agents and handoffs | LiveKit docs | Pipeline nodes and hooks | LiveKit docs | Agent speech and audio | LiveKit docs