is there any way livekit for speaker diarization and ignore background human voices. Just consider loudest speaker as the primary one and ignore all others. since now my agent is getting interrupted, and due to that STT is transcribing even those human voices which are not needed.

Thanks for reaching out, Developer. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

BVC suppresses. this is already done, can bvc be used with self hosted livekit model.

Thanks — I see you’ve already enabled BVC and want to know if it works with a self-hosted LiveKit deployment. Yes — BVC can be used with self-hosted LiveKit. The noise-cancellation models (including

BVC()

and

BVCTelephony()

) run in the agent process via the LiveKit Agents SDK, so they work the same in self-hosted deployments as in LiveKit Cloud. Key points and next steps • Enable the plugin in your agent code using the

livekit-plugins-noise-cancellation

package (example:

noise_cancellation.BVC()

for WebRTC,

noise_cancellation.BVCTelephony()

for SIP/8kHz). • Noise cancellation runs locally on the agent’s CPU, so ensure your agent host has sufficient CPU resources. • Install the noise-cancellation plugin in the agent environment where processing occurs. • Disable any other endpoint/SIP-level noise cancellation (for example, Krisp) to avoid double-processing. • No special server-side configuration is required in the self-hosted LiveKit server; the filtering is applied by the agent before audio is sent to STT. If you want, tell me: • Whether your agents are handling browser WebRTC callers or SIP/telephony callers, • Where the agent process runs (on-prem, cloud VM, container), and • Which STT provider you use. With that I can give a short, specific code snippet or resource sizing guidance for your environment. Sources: Choosing the right noise cancellation for your agent | LiveKit

livekit-plugins-noise-cancellation.

this is already enabled, but still agent is not able to ignore background voices.

Thanks — I understand you’ve already enabled

livekit-plugins-noise-cancellation

with BVC but background voices are still getting through and being transcribed. Let’s troubleshoot this together and narrow down the root cause. Quick checklist to confirm first • Which participant type are you processing? (Browser/WebRTC or SIP/telephony) • Which model are you actually using in your agent code:

noise_cancellation.BVC()

or

noise_cancellation.BVCTelephony()

? • Where does the agent process run (on-prem VM, container, cloud VM) and does it have plenty of CPU available? • Are there any other noise cancellation layers enabled (endpoint apps like Krisp, SIP trunk

krisp_enabled

, or host-level audio processing)? • Is the STT receiving the processed audio (filtered) or the raw audio? Where in your pipeline do you apply the filter? Immediate things to check and try 1. Confirm correct model for participant kind ◦ Use

BVC()

for full-bandwidth WebRTC callers and

BVCTelephony()

for narrowband 8kHz SIP calls. Using the wrong model reduces effectiveness. 1. Ensure no double-processing ◦ Disable any other noise suppression (Krisp, system-level filters, SIP-side NC). Double-processing can worsen results. 1. Verify filter placement in the pipeline ◦ Make sure the noise cancellation plugin is applied before audio is sent to your STT. If STT is receiving raw audio, BVC won’t help. 1. Resource check ◦ Noise cancellation runs locally on the agent’s CPU. If the agent host is resource-constrained, the model may underperform. Try running the agent on a machine with more CPU and see if suppression improves. 1. Test toggles to isolate behavior ◦ Compare three test runs and examine resulting audio/STT output: 1) No noise cancellation 2)

BVC()

(or

BVCTelephony()

) enabled 3) Any other NC (if previously active) enabled • This helps determine if BVC is being applied and whether it helps or hurts. 1. Audio characteristics that limit suppression ◦ If the primary speaker is very quiet or distant compared to background talkers, aggressive suppression may either fail or remove parts of the primary voice. Confirm the primary speaker is typically the loudest/closest source. If you can share any of the following, I can give exact next steps: • A short description of your pipeline (where you instantiate the noise filter and where STT ingests audio). • Which STT provider you use and whether you use speaker diarization at the STT level. • Whether the callers are WebRTC or SIP, and an example of the agent code that sets up `room_io`/audio input. If you prefer, paste a minimal snippet of how you apply

noise_cancellation

in your agent and I’ll review it for misconfiguration and suggest a precise fix. Sources: Choosing the right noise cancellation for your agent | LiveKit | Speechmatics STT plugin guide | LiveKit Documentation | Troubleshooting STT not picking up utterances | LiveKit | Agent session | LiveKit Documentation

import logging import os from datetime import datetime, timezone import time from pathlib import Path import json from dotenv import load_dotenv from livekit import ( rtc, api, ) from livekit.agents import ( Agent, AgentServer, AgentSession, JobContext, JobProcess, RunContext, ChatContext, cli, function_tool, room_io, metrics ) from livekit.agents import UserStateChangedEvent, AgentStateChangedEvent from livekit.plugins import ( openai, elevenlabs, silero, ) from livekit.plugins.turn_detector.multilingual import MultilingualModel from livekit.plugins import noise_cancellation from stt import build_stt, build_cloud_stt, build_local_stt from llm import build_llm, build_cloud_llm, build_local_llm from tts import build_tts from fino_rag import query_info, query_info_stub from fino_indexing import get_index from config import SYSTEM_PROMPT, execution_mode from mongo_push import push_to_mongo logger = logging.getLogger("finopayment_agent") # basic formatter logging.basicConfig( filename="app.log", level=logging.DEBUG, format="%(asctime)s [%(levelname)s] %(name)s - %(message)s", ) # make the noisy bits verbose for name in ( "livekit", "livekit.agents", "httpx", "openai", "elevenlabs", ): logging.getLogger(name).setLevel(logging.DEBUG) class Assistant(Agent): def __init__(self, system_prompt=SYSTEM_PROMPT) -> None: super().__init__( instructions=system_prompt, tools=[query_info if execution_mode == "local" else query_info_stub], ) async def on_enter(self): # await self.session.generate_reply() await self.session.say("Hello!, I am Purnima from Fino Payments Bank. Can I have a moment of your time?") server = AgentServer() def prewarm(proc: JobProcess): # load silero vad once per worker process proc.userdata["vad"] = silero.VAD.load( # activation_threshold=0.6, #Confidence threshold to classify audio as speech # min_silence_duration=0.3, # min_speech_duration=0.2, ) # warm up the retrieval index (loads embeddings, connects to qdrant) if execution_mode == "local": get_index() server.setup_fnc = prewarm async def write_transcript( ctx: JobContext, session: AgentSession, ): # This code runs when the agent task is shutting down current_date = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") reports_dir = Path("reports") reports_dir.mkdir(exist_ok=True) filename = reports_dir / f"transcript_{ctx.room.name}_{current_date}.json" try: # We use session.history.to_dict() to get the full chat log with open(filename, 'w', encoding='utf-8') as f: json.dump(session.history.to_dict(), f, indent=2, ensure_ascii=False) logger.info(f" Transcript saved to {filename}") return filename except Exception as e: logger.error(f"Failed to save transcript: {e}") return None async def push_session_to_mongo( end_time: float, start_time: float, ctx: JobContext, session: AgentSession, ): transcript_json = session.history.to_dict() duration = int(end_time - start_time) call_datetime = datetime.fromtimestamp(start_time, timezone.utc).strftime('%Y-%m-%d %H:%M:%S') recording_url = f"{os.getenv('EOS_ENDPOINT')}/{os.getenv('EOS_BUCKET_NAME')}/livekit/{ctx.room.name}.ogg" push_to_mongo(duration, call_datetime, transcript_json, recording_url) # async def log_usage(usage_collector: metrics.UsageCollector): # summary = usage_collector.get_summary() # logger.info(f"Usage: {summary}") @breezy-insurance-66772.rtc_session() async def entry( ctx: JobContext ): start_time = time.time() # add room info to all log entries ctx.log_context_fields = { "room": ctx.room.name, } if execution_mode == "local": stt_engine = build_local_stt() llm_engine = build_local_llm() else: stt_engine = build_cloud_stt() llm_engine = build_cloud_llm() # small hardcoded workaround to use cloud llm always, as we are evaluating gpt 4o mini # stt_engine = build_cloud_stt() # llm_engine = build_cloud_llm() # llm_engine = build_local_llm() tts_engine = build_tts() # usage_collector = metrics.UsageCollector() session = AgentSession( stt=stt_engine, llm=llm_engine, tts=tts_engine, max_endpointing_delay=0.2, # turn_detection=MultilingualModel(), turn_detection="vad", vad=ctx.proc.userdata["vad"], preemptive_generation=True, # min_interruption_duration=0.2, false_interruption_timeout=2.0, resume_false_interruption=True, ) @ctx.room.on("track_published") def on_track_published( publication: rtc.RemoteTrackPublication, participant: rtc.RemoteParticipant ): # Only subscribe to audio tracks from participants (not screen shares) if publication.kind == rtc.TrackKind.KIND_AUDIO: publication.set_subscribed(True) # Subscribe to audio tracks from the primary speaker only for p in list(ctx.room.remote_participants.values())[1] # Only first participant for pub in p.track_publications.values(): if pub.kind == rtc.TrackKind.KIND_AUDIO: pub.set_subscribed(True) # ctx.add_shutdown_callback(lambda: write_transcript(ctx=ctx, session=session)) # ctx.add_shutdown_callback(lambda: log_usage(usage_collector)) ctx.add_shutdown_callback(lambda: push_session_to_mongo(end_time=time.time(), start_time=start_time, ctx=ctx, session=session)) egress_req = api.RoomCompositeEgressRequest( room_name=ctx.room.name, audio_only=True, file_outputs=[ api.EncodedFileOutput( file_type=api.EncodedFileType.OGG, filepath=f"livekit/{ctx.room.name}.ogg", s3=api.S3Upload( bucket=os.getenv("EOS_BUCKET_NAME"), access_key=os.getenv("EOS_ACCESS_KEY"), secret=os.getenv("EOS_SECRET_KEY"), endpoint=os.getenv("EOS_ENDPOINT"), ), ) ], ) lkapi = api.LiveKitAPI() await lkapi.egress.start_room_composite_egress(egress_req) # await lkapi.aclose() # await ctx.wait_for_participant() try: # Start egress BEFORE the session to ensure it's ready to capture audio # try: # await lkapi.egress.start_room_composite_egress(egress_req) # logger.info("Started room composite egress for audio recording to S3.") # except Exception as e: # logger.error(f"Failed to start egress: {e}", exc_info=True) # continue; session can still run even if egress fails # Now start the session await session.start( agent=Assistant(system_prompt=SYSTEM_PROMPT), room=ctx.room, room_options=room_io.RoomOptions( audio_input=room_io.AudioInputOptions( noise_cancellation=noise_cancellation.BVC(), ), ) ) except Exception as e: logger.error(f"Failed to start session: {e}", exc_info=True) raise finally: try: await lkapi.aclose() logger.info("Closed lkapi.") except Exception as e: logger.debug(f"Failed to close lkapi: {e}", exc_info=True) if name == "__main__": cli.run_app(server) this is agent script running.

Thanks — I reviewed your agent script. I can see you instantiate

noise_cancellation.BVC()

in

room_io.AudioInputOptions

, which is the correct place to apply agent-side filtering. Let’s walk through targeted checks and quick fixes to find why background voices are still reaching your STT. What I’m responding to • You enabled

livekit-plugins-noise-cancellation.BVC()

in the agent but STT still transcribes background human voices. Immediate things to verify (quick checklist) • Participant type: Are callers WebRTC (browser) or SIP? If SIP/telephony, use

BVCTelephony()

instead of

BVC()

. • Double-processing: Confirm no other NC is enabled: ◦ Any endpoint-level tools (Krisp or OS-level NC)? ◦ Any SIP trunk setting like

krisp_enabled

? • Filter placement: Ensure the STT receives the processed audio. Confirm your STT ingest reads from the AgentSession audio path (session.stt) after

session.start()

applies the RoomOptions filter — not from a raw track copy or server-side egress that captures unfiltered audio. • CPU & performance: Is the agent process CPU-starved? BVC runs locally; under-provisioned CPU can reduce effectiveness. Code-specific notes and suggestions 1) Participant selection logic • You have this block that subscribes only the first remote participant: for p in list(ctx.room.remote_participants.values())[1] # Only first participant That subscribes only the first participant in the room object list — if callers or agents join in different orders this may not be the actual primary speaker. Make sure the participant you subscribe to is the one whose audio should go through BVC. Consider subscribing based on role/metadata or selecting the actively-speaking participant instead of list index. 2) Confirm BVC is applied to the same audio path used by STT • You set: room_options=room_io.RoomOptions( audio_input=room_io.AudioInputOptions( noise_cancellation=noise_cancellation.BVC(), ), ) This is correct when the AgentSession STT pulls audio from `session`’s audio input pipeline. Verify your

build_local_stt()

/

build_cloud_stt()

implementations consume the AgentSession audio stream (not a raw egress file or a different track). If STT is reading the room composite egress or an unfiltered track, it will get raw audio. 3) Egress vs live STT path • You start a room composite egress to record to S3/OGG. Egress captures the server-side mixed audio and may not include agent-applied filtering. Confirm whether STT is using live agent audio or the egress file: ◦ If STT uses the egress recording, that may be unfiltered. Try pointing STT to the AgentSession audio directly (live) or ensure egress uses the filtered stream. 4) Use BVCTelephony for 8kHz SIP calls • If callers are PSTN/SIP (8 kHz), switch to

noise_cancellation.BVCTelephony()

dynamically: filter = noise_cancellation.BVCTelephony() if participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP else noise_cancellation.BVC() 5) Debugging steps to confirm behavior • Log whether filter is active at session start: ◦ After

await session.start(...)

, add a debug log indicating which noise filter instance is in use. • Capture and inspect a short processed audio sample that is fed to STT: ◦ Write out a 10–15s clip from the agent pipeline (post-filter) and from the raw track, and compare audibly or with a spectrogram to confirm background voices are suppressed in the processed clip. • Run isolation tests: a. No NC (remove

noise_cancellation

), record STT output. b.

BVC()

enabled, record STT output. c.

BVCTelephony()

if SIP, record STT output. Compare transcripts to confirm which stage is most effective. 6) VAD and interruption settings • You use

turn_detection="vad"

and a Silero VAD instance. If VAD thresholds are too aggressive or permissive, it may mark background speech as valid turns. Try adjusting VAD activation thresholds or min speech duration to better detect the primary voice versus background chatter. 7) Resource sizing • If your agent runs in a constrained container or VM, try launching it on a beefier machine temporarily to check if suppression quality improves. If you’re willing, please share one of the following so I can give exact code-level fixes: • Is the incoming audio WebRTC or SIP? • The

build_local_stt()

/

build_cloud_stt()

function (the part that connects STT to session audio). • Whether STT is fed live from AgentSession or from the egress/recording file. If you paste the STT connection snippet I’ll: • Verify the audio path is post-BVC, • Suggest an updated minimal code sample to ensure STT reads the filtered audio, • And provide a short test snippet to produce a post-filter WAV so you can confirm suppression locally.

Incoming audio is SIP? def build_cloud_stt(): model = os.getenv("OPENAI_STT_MODEL", "gpt-4o-transcribe") language = os.getenv("STT_LANGUAGE", "en") logger.info(f"using openai stt: model={model}, language={language}") return openai.STT( model=model, language=language, # task="transcribe", ) STT is fed live from AgentSession

Thanks — this is helpful. You confirmed incoming audio is SIP and STT is fed live from AgentSession. I reviewed your code and a few likely causes and targeted fixes to get BVC working for SIP calls. What I’m responding to • You’re using SIP callers and

BVC()

is enabled, but background voices still reach your live STT (OpenAI). For SIP you should use the telephony-optimized model, confirm the filter is on the STT audio path, and rule out other causes. Actionable fixes and checks (in order) 1) Use BVCTelephony for SIP • Change the noise cancellation to

BVCTelephony()

for SIP participants. Replace your current static

BVC()

with telephony-aware selection when starting the session:

# if all incoming are SIP, use BVCTelephony()
room_options=room_io.RoomOptions(
    audio_input=room_io.AudioInputOptions(
        noise_cancellation=noise_cancellation.BVCTelephony(),
    ),
)

• If a mix of SIP and WebRTC exists, detect participant kind and choose the filter per participant (or per session) dynamically. 2) Confirm STT is receiving the post-filter audio • You said STT is fed live from AgentSession — ensure the agent’s audio input pipeline (the RoomOptions you set when calling

session.start(...)

) is the same pipeline your STT implementation consumes. • Verify

build_cloud_stt()

reads from the AgentSession audio (it should, since you pass

stt=stt_engine

into AgentSession). If you’ve modified STT wiring, confirm no alternate path (egress, raw track) is used. 3) Ensure no double-processing (disable other NC) • Make sure SIP trunks, endpoints, or OS-level tools (Krisp) are not applying noise suppression. Double-processing can reduce BVC effectiveness. 4) Check agent CPU and performance • BVCTelephony runs locally in the agent process. If the agent host is CPU constrained, suppression can underperform. Try running the agent on a machine with more CPU to see if suppression quality improves. 5) Confirm subscription to the correct participant • Your code subscribes to only the first remote participant in room list: for p in list(ctx.room.remote_participants.values())[1] • The SIP caller you expect may not be the first in that list. Ensure you subscribe to the actual SIP participant (match by metadata or participant identity) or subscribe to all audio and apply filtering to the agent input path you expect. 6) VAD / interruption settings • You use Silero VAD. If VAD thresholds consider background speech valid, STT may still be triggered. Try tuning Silero VAD activation_threshold / min_speech_duration to be stricter so background chatter isn’t treated as a speech turn. 7) Debugging tests to isolate the problem • Short A/B transcript test: 1) Run with no NC and record STT results. 2) Run with

BVCTelephony()

and record STT results. 3) If available, run with endpoint NC enabled (only to compare). Compare transcripts to confirm which reduces background transcriptions. • Capture post-filter audio sample: ◦ Temporarily write a 10–15 second WAV from the AgentSession audio pipeline (post-filter) and compare it against the raw track/egress file. This proves what STT actually hears. • Log filter use at startup: ◦ After

await session.start(...)

, log which

noise_cancellation

class instance you passed so you know BVCTelephony is active. Suggested minimal edits (example) • Replace your session start block with telephony filter:

await session.start(
    agent=Assistant(system_prompt=SYSTEM_PROMPT),
    room=ctx.room,
    room_options=room_io.RoomOptions(
        audio_input=room_io.AudioInputOptions(
            noise_cancellation=noise_cancellation.BVCTelephony(),
        ),
    )
)
<http://logger.info|logger.info>("Started session with BVCTelephony noise cancellation")

If you want, next I can: • Review your

build_local_stt()

if you also use a local STT path, or • Provide a short snippet to record a post-filter WAV from the AgentSession pipeline so you can listen/inspect what STT receives. Tell me which you prefer and paste (if possible) the participant metadata or how SIP participants are identified in your environment — I can then suggest a robust participant selection approach so you always apply BVCTelephony to actual SIP callers.