https://livekit.io logo
Join Slack
Powered by
What is the best way to improve my voice agent in ...
# ask-ai
f
What is the best way to improve my voice agent in very noisy conditions or with interfering background voices?
t
Thanks for reaching out, Fabian. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m glad you asked. Improving a voice agent in very noisy conditions or where other voices interfere usually works best by combining three things: enhanced noise cancellation, speaker diarization (so the agent focuses on the primary speaker), and robust turn detection/VAD tuning. Below I’ll outline practical steps you can take and things to try. What I’m responding to: your question about improving an agent’s performance in very noisy environments and with interfering background voices. 1) Enable enhanced noise cancellation • Use LiveKit’s built-in noise cancellation plugins (for example, the BVC or BVCTelephony options) in your RoomOptions so audio sent to STT/LLM is already cleaned. • This reduces steady background noise and isolated non-speech sounds, which helps both transcription accuracy and the agent’s downstream understanding. 2) Use speaker diarization and suppress background speakers • Enable diarization on your STT model or wrap your STT with a MultiSpeakerAdapter so the system identifies distinct speakers. • Configure the flow to only forward the primary speaker’s transcript to the LLM (or mark other speakers as background). This prevents background conversations from being treated as the user’s input. • If you need transcripts for other speakers (for logging or moderation), keep them separate and clearly labeled instead of mixing them into the agent’s input. 3) Tune turn detection and VAD for noisy environments • Use a robust turn-detection model (for example, a MultilingualModel-style detector) that is designed for real-world noise. • Adjust VAD parameters like 
min_speech_duration
and 
min_silence_duration
to avoid false starts from noise bursts: ◦ Increase 
min_speech_duration
slightly to ignore very short noise blips. ◦ Increase 
min_silence_duration
so the system doesn’t cut off when background sounds momentarily drop. • The goal is to reduce false positives (noise detected as speech) and false negatives (real speech missed). 4) Combine strategies for best effect • Pipeline example: noise cancellation → VAD/turn detection → diarization → STT → LLM. • Noise cancellation cleans raw audio first; VAD/turn detection prevents sending noise-only segments; diarization ensures only the intended speaker’s text reaches the agent. 5) Other practical tips • If possible, ask users to use headsets or closer microphone placement — hardware gains are often the most effective. • If background voices are predictable (e.g., common hold-music, repeating announcements), you can add filters or keyword-based suppression for those known noises. • Consider confidence thresholds from the STT: if transcription confidence is low, prompt for a short confirmation before the agent acts. If you’d like, I can: • Suggest specific parameter values for 
min_speech_duration
/ 
min_silence_duration
based on your use case (IVR vs. in-person kiosk vs. group environment). • Show a short code snippet of how to wire noise cancellation, diarization, and the turn detector together for your setup. • Help you choose between different STT diarization-capable models if you tell me which providers you’re using (Deepgram, Speechmatics, etc.). Which of those follow-ups would be most helpful right now? Sources: LiveKit Platform | Build, run, and observe voice AI agents | LiveKit | Speech-to-text (STT) models overview | LiveKit Documentation | Turns overview | LiveKit Documentation
f
Which noise cancellation options does LiveKit have?
t
Thanks — good question. I’m responding to your request for which noise cancellation options LiveKit offers so you can pick the best fit for noisy environments. LiveKit provides several noise-cancellation choices you can use depending on participant type and deployment: • BVC (Background Voice Cancellation) ◦ A general-purpose noise cancellation filter provided by the noise_cancellation plugin. Good for typical WebRTC audio and common background noise scenarios. • BVCTelephony ◦ A variant of BVC optimized for narrow-band telephony/SIP audio. Use this when participants connect via SIP to get better suppression for telephone audio. • Krisp (AI-powered noise cancellation) ◦ Advanced, AI-driven noise suppression that works at the media server / trunk or participant level. Can be enabled for inbound or outbound SIP calls (via a 
krisp_enabled
parameter) and is supported in LiveKit Cloud for enhanced server-side noise removal. Best when you need stronger real-time suppression of background noise and echo. • Bypass native voice processing (iOS) ◦ On iOS you can disable the platform’s built-in voice processing (echo/noise suppression) using 
bypassVoiceProcessing
in 
LiveKitClient.initialize()
. This yields raw audio so you can apply LiveKit’s noise cancellation or your custom processing pipeline instead of the native iOS processing. When to choose which: • Use BVC for general WebRTC scenarios where you need a lightweight, built-in filter. • Use BVCTelephony for SIP/telephony callers to handle narrow-band audio characteristics. • Use Krisp when you need stronger AI-based suppression (e.g., very noisy environments, persistent background voices or echo). • Use bypassVoiceProcessing if you need control over raw audio input (for custom models or to avoid double-processing by iOS). If you’d like, I can: • Show a short code example wiring one of these (BVC or Krisp) into RoomOptions or trunk config. • Recommend specific settings or values (e.g., when to prefer Krisp vs. BVC) based on whether your users connect via WebRTC, SIP, or mobile apps. Which platform or call type are you using (WebRTC participants, SIP trunk, or mobile app)? That will help me give a concrete example. Sources: Agent session | LiveKit Documentation | Turns overview | LiveKit Documentation | Using Noise Cancellation for SIP and WebRTC Participants | sip | livekit/livekit_composite | DeepWiki | Agents telephony integration | LiveKit Documentation
f
Don't you have another option from ai-coustics?
t
Yes — LiveKit includes ai-coustics noise cancellation in LiveKit Cloud. I’m responding to your question about whether there’s an ai-coustics option beyond the previously listed choices. The ai-coustics filters available in LiveKit are: • BVC (Background Voice Cancellation) — for general WebRTC audio. It suppresses background noise and non-human sounds while preserving speech. • BVCTelephony — a telephony-optimized variant tuned for narrow-band SIP audio; it suppresses background noise and interfering voices for SIP/telephone callers. These are the ai-coustics-powered enhanced noise-cancellation filters you can enable. Use BVC for typical WebRTC participants and BVCTelephony when handling SIP/telephony connections. If you’d like, I can: • Show a short code example for applying BVC vs BVCTelephony based on participant type. • Recommend when to prefer Krisp (AI-powered suppression at trunk/participant level) versus the ai-coustics filters for very noisy environments. Which would be most useful for your setup: a code example or a recommendation based on whether you use WebRTC, SIP, or mobile apps? Sources: Using Noise Cancellation for SIP and WebRTC Participants | Agents telephony integration | LiveKit Documentation | Turns overview | LiveKit Documentation | Agent session | LiveKit Documentation
f
Yes, please recommend when to use ai-coustics and when to use Krisp.
t
Thanks — good question. Below I’ll compare ai-coustics (BVC / BVCTelephony) and Krisp, and give clear guidance on when to choose each so you can pick the best fit for your voice agent. What I’m responding to: your request for a recommendation on when to use LiveKit’s ai-coustics filters (BVC / BVCTelephony) versus Krisp. Summary recommendation • Use ai-coustics (BVC / BVCTelephony) when you want low-latency, server-side noise suppression that’s optimized for typical WebRTC and SIP call patterns and when you want built-in LiveKit Cloud integration with minimal setup. • Use Krisp when you need stronger AI-powered suppression of persistent, complex background noises or echo (for example, loud cafes, machinery, overlapping conversations), or when you want the additional trunk/participant-level options Krisp provides for SIP workflows. Detailed guidance • ai-coustics (BVC / BVCTelephony) ◦ Best for: Most WebRTC use cases and standard telephony/SIP calls. ◦ Strengths: ▪︎ Built into LiveKit Cloud as Enhanced noise cancellation. ▪︎ Low latency and tuned for real-time voice interactions. ▪︎ BVC is optimized for general WebRTC audio and non-human background noise. ▪︎ BVCTelephony is tuned for narrow-band telephony/SIP audio and better suppresses interfering voices for phone callers. ▪︎ Simple to enable via RoomOptions or participant selection and works well combined with diarization/VAD. ◦ Choose ai-coustics when you need reliable, efficient suppression with minimal extra configuration and you expect typical background noise (traffic, HVAC, indistinct ambient sounds). • Krisp (AI-powered noise cancellation) ◦ Best for: Very noisy, real-world environments or situations with persistent/interfering human voices and echo where stronger suppression is required. ◦ Strengths: ▪︎ Advanced AI models that remove complex background noise and reduce echo more aggressively. ▪︎ Can be enabled per SIP trunk (inbound) or per participant (outbound) and is supported at the LiveKit Cloud media server level. ▪︎ Useful when telephony participants frequently call from uncontrolled noisy places (call centers with loud floors, busy public spaces, industrial environments). ◦ Consider Krisp when you see transcription or turn-detection errors caused by persistent background voices or when ai-coustics alone doesn’t give the clarity you need. Practical decision criteria • If your users are mostly WebRTC participants in typical offices/homes: start with BVC. • If your users call via SIP/telephony: use BVCTelephony for narrow-band optimization. • If audio still suffers from overlapping speakers, loud music, machinery, or echo after ai-coustics: enable Krisp for those flows (inbound trunk or participant-level). • If you need minimal latency and a straightforward built-in solution: prefer ai-coustics first, then escalate to Krisp only where necessary. Example workflows • WebRTC app in-office/home calls: enable BVC, plus diarization + tuned VAD. • SIP-based contact center with typical call noise: enable BVCTelephony on the SIP path. • Contact center or public kiosk with very noisy surroundings: enable BVCTelephony and enable Krisp on the SIP trunk or participant for extra suppression. • Mobile apps where iOS native processing might interfere: consider disabling native processing (
bypassVoiceProcessing
) and use ai-coustics or Krisp depending on noise level. Would you like a short code example that: • Shows how to select BVC vs BVCTelephony based on participant type, or • Shows how to enable Krisp for a SIP trunk or participant? Tell me which platform (WebRTC, SIP/trunk, or mobile) you want an example for and I’ll provide the snippet.
f
Is there a Voice Focus option as well?
t
Thanks — I’m answering your question about whether LiveKit has a "Voice Focus" option. LiveKit does not offer a "Voice Focus" feature (that name is specific to Cisco/Webex). Instead, LiveKit provides these noise-suppression options you can use to achieve similar results: • ai-coustics (BVC / BVCTelephony) ◦ BVC: background-voice cancellation for general WebRTC audio (preserves speech, removes non‑speech noise). ◦ BVCTelephony: telephony-tuned BVC for narrow-band SIP callers; better at suppressing interfering voices on phone audio. • Krisp (AI-powered noise cancellation) ◦ Stronger AI suppression of persistent, complex background noise and echo. Can be enabled per SIP trunk or per participant and is available at the LiveKit Cloud media-server level. • Bypass native voice processing (iOS) ◦ Disable iOS built-in voice processing with 
bypassVoiceProcessing
to get raw audio and apply LiveKit’s filters or your own processing. Which to pick: • For typical WebRTC calls: start with BVC. • For SIP/telephone callers: use BVCTelephony. • If noise, overlapping voices, or echo remains a problem: enable Krisp for those flows. If you’d like, I can: • Show a short code snippet that applies BVC vs BVCTelephony depending on participant type, or • Show how to enable Krisp on a SIP trunk/participant. Which example would help you most? Sources: Using Noise Cancellation for SIP and WebRTC Participants | Agents telephony integration | LiveKit Documentation
7 Views
Open in Slack
PreviousNext
Powered by