how livekit background voice cancelation works for agents? for self hosted/hybrid/cloud options

Thanks for reaching out, Lasha. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

import { config } from 'dotenv'; import { resolve, dirname } from 'node:path'; import { fileURLToPath } from 'node:url'; // Load .env from project root const __dirname = dirname(fileURLToPath(import.meta.url)); config({ path: resolve(__dirname, '..', '..', '..', '.env') }); import { defineAgent, type JobContext, type JobProcess, cli, WorkerOptions, voice, } from '@alert-easter-19258/agents'; import * as silero from '@alert-easter-19258/agents-plugin-silero'; import * as livekit from '@alert-easter-19258/agents-plugin-livekit'; import * as elevenlabs from '@alert-easter-19258/agents-plugin-elevenlabs'; import { BackgroundVoiceCancellation } from '@alert-easter-19258/noise-cancellation-node'; import { searchKnowledgeBase } from './tools/knowledge-base.js'; import { connectToMongoDB, getAgentInstructions, getAgentConfig } from './services/mongodb.js'; import { createSessionContext, initSession, logUserSpeech, logAgentResponse, logSessionEnd, markAgentSpeaking, collectMetrics, } from './hooks/logging.js'; import type { DispatchMetadata, AgentConfig, TtsProvider } from './types/index.js'; const DEFAULT_INSTRUCTIONS = `You are an AI conversational agent representing Evolvexe, a full-scale BPO and technical support outsourcing company based in Georgia. Your primary role is to: - Provide accurate, structured, and trustworthy information about Evolvexe - Support sales and lead generation without being aggressive - Guide interested users toward human contact when appropriate - Operate strictly within approved knowledge boundaries You must only answer based on the approved Evolvexe knowledge base. If information is not available, do not guess. TOOL USAGE: - ONLY use the searchKnowledgeBase tool when the customer asks a specific question that requires looking up information. - Do NOT call any tools for greetings, small talk, or general conversation. - Synthesize information from the knowledge base into natural, conversational responses. - Do not read out article titles or metadata — just answer naturally. TONE OF VOICE AND COMMUNICATION STYLE: - Professional, confident, and friendly - Clear and structured responses - Business-oriented, not marketing hype - Helpful and consultative, not pushy - Calm and neutral in all situations - Avoid over-selling, informal slang, emotional or opinionated language INFORMATION SCOPE — ALLOWED: - Explain Evolvexe's services, industries, and capabilities - Describe technical support and AI solutions at a semi-technical level - Position Evolvexe competitively using general advantages - Explain why pricing is cost-effective without giving numbers - Describe Georgia as a business-friendly, cost-efficient location - Explain recruitment and onboarding processes at a high level - Offer meetings and collect lead details INFORMATION SCOPE — RESTRICTED (MUST NOT): - Invent or assume facts - Provide internal KPIs, SLAs, CSAT, AHT, FCR, QA scores - Share exact prices, margins, salaries, or financials - Disclose confidential client details - Give legal, tax, or regulatory advice - Discuss internal company politics or contracts If restricted topics arise, redirect politely to management contact. SALES AND LEAD HANDLING: When the user shows interest in services, asks about pricing, SLAs, or contracts, requests custom solutions, or asks for comparisons or deeper details — offer a meeting. - Meetings are available every Wednesday from 1700 1800 Georgian time - Offer options: ask the user to leave their email or phone number, or suggest contacting Evolvexe via corporate email - Clearly state that a management representative will follow up - Never confirm a deal or make commitments PRICING GUARDRAILS: - You may say pricing is customized based on project scope and complexity - You may say Evolvexe is more cost-effective than providers in the USA and EU - You may say Georgia's tax and regulatory environment supports competitive pricing - You must NOT give numbers, estimate costs, provide ranges, or say "starting from" prices - Always redirect pricing discussions to a meeting AI AND TECHNOLOGY EXPLANATIONS: - Use semi-technical explanations - Explain AI workflows, automation logic, and human + AI collaboration - Avoid model names, training datasets, and proprietary technical details - State that AI tools are white-label, offered to clients as products, and enhance humans rather than replace them COMPLIANCE AND DATA PROTECTION: - State that Evolvexe operates under Georgian legislation aligned with GDPR - Mention existence of a dedicated data protection division - Do NOT interpret laws, provide compliance advice, or guarantee legal outcomes - For compliance questions, offer a management call COMPETITOR COMPARISON: - Remain neutral at all times - Do not name or compare competitors directly - Do not claim exclusivity or market dominance - Focus on Evolvexe's strengths only REFUSAL AND REDIRECTION (MANDATORY): If a user asks for restricted information: 1. Politely state that you cannot provide that detail 2. Explain that it requires a management discussion 3. Offer to schedule a meeting or collect contact details Example: "This information is usually discussed directly with our management team. I'd be happy to help you schedule a call or pass your contact details to Evolvexe." SAFETY AND HALLUCINATION PREVENTION: - If unsure, say: "This information is not available in my current scope." - Never guess, never fabricate, never contradict the knowledge base LANGUAGE: - Default language: English - Maintain professional international tone - If language is unsupported, politely ask to switch to English`; // ── TTS builder: maps config to a TTS instance or Inference string ── function buildTts(ttsConfig: { provider: TtsProvider; model: string; voiceId: string }): string | elevenlabs.TTS { const { provider, model, voiceId } = ttsConfig; if (provider === 'elevenlabs') { // Config (UI/DB) takes priority over env return new elevenlabs.TTS({ voiceId: voiceId || process.env.ELEVENLABS_VOICE_ID || 'hpp4J3VqNfWAUOO0d1Us', model, }); } // All other providers → LiveKit Inference string: "provider/model:voiceId" return `${provider}/${model}:${voiceId}`; } export default defineAgent({ prewarm: async (proc: JobProcess) => { const t0 = Date.now(); // Pre-load VAD ONNX model + MongoDB connection in parallel. // VAD is created per-session with dynamic config, but prewarming caches the model weights. const [vad] = await Promise.all([ silero.VAD.load(), connectToMongoDB(), ]); proc.userData.vad = vad; console.log(

[Agent] Prewarmed in ${Date.now() - t0}ms — VAD model cached, MongoDB connected

); }, entry: async (ctx: JobContext) => { // Wait for the agent to be connected to the room await ctx.connect(); // Extract metadata from the dispatch/job (embedded in token's RoomAgentDispatch) const rawMetadata = ctx.job.metadata ?? '{}'; let metadata: DispatchMetadata; try { metadata = JSON.parse(rawMetadata); } catch { console.error('[Agent] Failed to parse dispatch metadata:', rawMetadata); metadata = { channel: 'web', user_id: 'unknown', session_id:

session_fallback_${Date.now()}

, }; } const { channel, user_id, session_id } = metadata; const roomName = ctx.room.name ?? 'unknown_room'; console.log(

[Agent] Joined room: ${roomName} | Channel: ${channel} | User: ${user_id} | Session: ${session_id}

); // Fetch instructions and config fresh from MongoDB every session (parallel) let instructions = DEFAULT_INSTRUCTIONS; const [dbInstructions, agentConfig] = await Promise.all([ getAgentInstructions().catch(() => null), getAgentConfig(), ]); if (dbInstructions) instructions = dbInstructions; const { stt, llm, tts, voice: voiceCfg } = agentConfig; console.log(

[Agent] Config: STT=${stt.model} | LLM=${llm.model} | TTS=${tts.provider}/${tts.model} | VAD silence=${voiceCfg.minSilenceDuration}ms | Endpointing=${voiceCfg.minEndpointingDelay}-${voiceCfg.maxEndpointingDelay}ms

); const sessionContext = createSessionContext(session_id, roomName, channel, user_id); // Create session document in MongoDB (fire-and-forget) const modelsUsed = { stt: stt.model, llm: llm.model, tts:

${tts.provider}/${tts.model}

, }; initSession(sessionContext, { models_used: modelsUsed }).catch((err) => console.error('[Logging] Failed to create session:', err) ); // Build VAD per-session with configured minSilenceDuration (ONNX model already cached from prewarm) const sessionVad = await silero.VAD.load({ minSilenceDuration: voiceCfg.minSilenceDuration, }); const agent = new voice.Agent({ instructions, // tools: { searchKnowledgeBase }, // temporarily disabled }); const session = new voice.AgentSession({ vad: sessionVad, stt: stt.model, llm: llm.model, tts: buildTts(tts), turnDetection: voiceCfg.turnDetectorEnabled ? new livekit.turnDetector.MultilingualModel() : undefined, voiceOptions: { preemptiveGeneration: voiceCfg.preemptiveGeneration, minEndpointingDelay: voiceCfg.minEndpointingDelay, maxEndpointingDelay: voiceCfg.maxEndpointingDelay, allowInterruptions: voiceCfg.allowInterruptions, minInterruptionDuration: voiceCfg.minInterruptionDuration, minInterruptionWords: voiceCfg.minInterruptionWords, }, }); session.on(voice.AgentSessionEventTypes.UserInputTranscribed, (ev) => { if (ev.isFinal && ev.transcript) { logUserSpeech(sessionContext, ev.transcript).catch((err) => console.error('[Logging] Failed to log user speech:', err) ); } }); session.on(voice.AgentSessionEventTypes.ConversationItemAdded, (ev) => { const item = ev.item; if ('role' in item && item.role === 'assistant') { const text = item.textContent; if (text) { logAgentResponse(sessionContext, text).catch((err) => console.error('[Logging] Failed to log agent response:', err) ); } } }); // Track agent state → 'speaking' for total TTFV measurement session.on(voice.AgentSessionEventTypes.AgentStateChanged, (ev) => { if (ev.newState === 'speaking') { markAgentSpeaking(sessionContext); } }); // Collect per-component latency metrics (EOU, LLM, TTS) session.on(voice.AgentSessionEventTypes.MetricsCollected, (ev) => { collectMetrics(sessionContext, ev.metrics); }); session.on(voice.AgentSessionEventTypes.Close, () => { console.log(

[Agent] Session closed: ${session_id}

); logSessionEnd(sessionContext).catch((err) => console.error('[Logging] Failed to log session end:', err) ); }); // Start the voice agent session with background voice cancellation await session.start({ agent, room: ctx.room, inputOptions: { noiseCancellation: BackgroundVoiceCancellation(), }, }); // Greet the user session.generateReply({ instructions: 'Greet the user warmly and ask how you can help them today. Do not use any tools for this greeting.', }); console.log(

[Agent] Session started, greeting sent for session: ${session_id}

); }, }); // Register with explicit agent name and start the CLI cli.runApp( new WorkerOptions({ agent: fileURLToPath(import.meta.url), agentName: process.env.AGENT_NAME || 'customer-service', }) ); is this snippet correct? We have agent worker service deployed on railway

Thanks — this looks like a solid, well-structured agent implementation. I’ll walk through the key areas I checked, call out a few small issues or things to verify for Railway deployment, and suggest a couple of safety checks and follow-up questions so we can pin down any problems you might see in practice. What I checked and why it looks correct • Environment loading: You load .env from project root using

dotenv

and file URL utilities — that approach is fine for local and most containerized environments, as long as the .env path exists in Railway build/runtime. • Prewarm and session setup: You prewarm the VAD and MongoDB connection, then create per-session VAD with configured

minSilenceDuration

— that’s correct and efficient. • Agent/Session construction: You create

voice.Agent

and

voice.AgentSession

with STT/LLM/TTS config and VAD, plus event handlers for transcription, conversation items, metrics, and session lifecycle — this follows expected patterns and will surface useful logs. • Background voice cancellation: You enable it via

inputOptions: { noiseCancellation: BackgroundVoiceCancellation() }

when starting the session — that is the correct place to apply agent-side BVC so audio is filtered before STT/agent processing. • Per-channel logic and TTS builder: The

buildTts

function and dynamic selection of models/providers looks consistent with how the agent should initialize TTS for sessions. • Telephony vs WebRTC: You already plan to use telephony-optimized filters (

BVCTelephony()

) when appropriate — good to match filter type to audio bandwidth. Small issues & things to verify • Escaped identifiers in your pasted snippet: ◦ I see

DEFAULTINSTRUCTIONS

,

DEFAULTINSTRUCTIONS

backslashes, and

DEFAULTINSTRUCTIONS

style escaping in the snippet you pasted. Ensure in your actual source files the identifier names are plain (e.g.,

DEFAULT_INSTRUCTIONS

) and not escaped by copy-paste/markdown artifacts. •

__dirname

declaration: ◦ You recreate

__dirname

as

const __dirname = dirname(fileURLToPath(import.meta.url));

— that’s correct for ESM, but ensure you’re using Node with ESM and not mixing CommonJS. • BackgroundVoiceCancellation import and use: ◦ You import

BackgroundVoiceCancellation

from

@LiveKit/noise-cancellation-node

and call it with no args. Confirm that

BackgroundVoiceCancellation()

is the intended factory function and that any optional config (if supported) is not required in your environment. • Tools commented out: ◦

tools: { searchKnowledgeBase }

is commented out. If you plan to enable the knowledge-base tool later, remember to re-add it and ensure permission/credentials are available. • Error handling around

getAgentConfig()

and `getAgentInstructions()`: ◦ You catch

getAgentInstructions()

failure but not

getAgentConfig()

— consider wrapping

getAgentConfig()

with a catch and default fallback to avoid runtime throws. • MongoDB connection prewarm: ◦ You await

connectToMongoDB()

as part of the Promise.all but only destructure

[vad]

. This still calls the function, but if you intended to capture the DB client, destructure both values or rely on

connectToMongoDB()

side effects. Current code is valid if connection is handled internally and stored globally. • Railway specifics — filesystem and environment: ◦ Railway may not persist a local

.env

file in production. Ensure you set env variables via Railway’s environment variable UI instead of relying on .env at runtime. ◦ Railway ephemeral filesystem: if your prewarm caches models on disk, verify the path and persistence expectations — cached files are ephemeral across restarts. • CPU / resource sizing: ◦ BVC and VAD consume CPU. On Railway, ensure the selected plan/container has sufficient CPU for real-time processing; otherwise you may see increased latency or dropped audio. • Logging and sensitive data: ◦ Avoid logging secrets. The code logs model names — that’s OK — but ensure environment keys (API keys, DB URIs) are not logged anywhere. Practical checks to run now 1. Start locally with the same NODE_ENV and env variables Railway will provide to confirm the code runs and BVC initializes without error. 2. On Railway: ◦ Add required env vars in Railway’s project settings (DB URI, API keys, AGENT_NAME, etc.). ◦ Ensure your service has adequate CPU/memory for real-time audio processing. 1. Watch logs for these startup signs: ◦ "[Agent] Prewarmed..." (VAD cached, MongoDB connected) ◦ "[Agent] Joined room: ..." after token/dispatch ◦ "[Agent] Session started, greeting sent..." after

session.start

and

generateReply

1. If you see BVC-specific errors, capture the full error text and we can debug config vs runtime issues. Questions to help me confirm fully • Are you seeing any specific runtime errors on Railway (startup failures, missing module errors, or runtime exceptions)? If yes, paste the exact log lines. • Are your participants primarily SIP (telephone) or WebRTC? I can confirm whether to call

BVCTelephony()

instead or change selection logic. • Do you want a small example snippet showing how to use

BVCTelephony()

vs

BVC()

dynamically per participant type? • Do you cache model files to disk during prewarm, or do you rely purely on in-memory downloads? (Railway ephemeral storage matters if you expect persisted caches.) If you share any Railway logs or confirm the participant types, I’ll give a targeted checklist or a short patch for anything that needs changing.

is there any way to verify that BVC actually started and is orking?

Thanks — I’ll walk you through concrete, practical ways to verify that BVC actually started and is working in your Railway-deployed agent. I’ll call out quick checks you can add to your code, runtime signals to watch in logs/metrics, and a simple audio test you can run. What I’m answering: how to confirm BVC initialization and runtime operation for your agent service 1) Add explicit startup logging (best first check) • When you create the filter, log that you initialized it and any returned metadata: ◦ Example: log right after

BackgroundVoiceCancellation()

is created: ▪︎ "BVC initialized: BackgroundVoiceCancellation() called" ◦ If the factory returns an object, log its type or properties. • Place a try/catch around the creation so you log any error immediately: ◦ On success: log the instance and the session/room id. ◦ On failure: log the full error stack so you can see missing modules or permission issues. 2) Observe runtime logs during session.start • Confirm these lines appear in your logs: ◦ "[Agent] Prewarmed..." (VAD cached, DB connected) ◦ "[Agent] Session started, greeting sent..." (session started) • Add an extra log where you pass

inputOptions

so you see a line like: ◦ "Starting session with noiseCancellation=BackgroundVoiceCancellation" • If you see initialization logs and no errors, BVC was at least created and attached to the session. 3) Monitor CPU usage and process metrics • BVC runs on agent CPU. While it’s low-latency, enabling it increases CPU during audio processing. • Compare CPU while processing test audio with BVC enabled vs disabled. A measurable increase during voice activity indicates the filter is running. • On Railway: ◦ Ensure the container shows non-idle CPU during calls. ◦ If you have app-level metrics, log CPU at session start and during active voice frames. 4) Use VAD / pipeline metrics to confirm audio flows through the pipeline • You already wire VAD and have a MetricsCollected handler. Use those events to ensure audio frames are being processed. • Track counts/timestamps for: ◦ VAD voice frames detected ◦ STT requests sent ◦ Any per-frame latency • If frames are processed and STT calls are being made only for primary speech, that’s evidence BVC is filtering as expected. 5) Behavioral audio tests (most reliable functional check) • Prepare a controlled test with a primary speaker + background speaker/noise. • Run 3 runs: a. No cancellation (disable BVC) — observe STT transcripts; background voice should appear. b. With BVC enabled — background voice should be suppressed from transcript. c. With frontend Krisp + agent-BVC both enabled — you may see degraded audio; this confirms double-processing issues. • If transcripts show background speech in (1) but not in (2), BVC is working. 6) Add short diagnostic audio capture (optional, for deeper debugging) • Capture a short sample of the pre- and post-BVC audio buffers (small, encrypted, and stored securely) to compare what the STT receives vs original input. • Use only in tests, and ensure you follow data protection rules. 7) Avoid double-processing • Confirm

krisp_enabled

is false on SIP trunks if using agent-side BVC. Double-processing can reduce quality and produce wrong STT results. Practical code snippets to add • Add an init log and safe creation:

let noiseFilter;
try {
  noiseFilter = BackgroundVoiceCancellation();
  console.log('[Agent] BVC initialized', { filter: !!noiseFilter });
} catch (err) {
  console.error('[Agent] Failed to init BVC', err);
}
...
await session.start({
  agent,
  room: ctx.room,
  inputOptions: {
    noiseCancellation: noiseFilter,
  },
});
console.log('[Agent] session.start called with BVC:', !!noiseFilter);

What to check in Railway logs / runtime • Look for the BVC init log and session.start logs. • Check for runtime errors related to the noise-cancellation module. • Watch CPU consumption during active calls and compare to a baseline without BVC. If you want, I can: • Provide a short test script/instructions you can run to generate the three audio runs (no BVC, BVC, double-processed) and compare STT output. • Show a minimal snippet that captures pre/post audio frames for a one-second test (securely) so you can audibly confirm filtering. Which would help most: a test script for STT comparison, or the minimal pre/post capture snippet? And are your participants SIP, WebRTC, or both? Sources: Choosing the right noise cancellation for your agent | LiveKit | VAD Metrics | LiveKit Documentation