https://livekit.io logo
Join Slack
Powered by
We have two silero vad instance created per entryp...
# ask-ai
s
We have two silero vad instance created per entrypoint() function in our python livekit agent. The first one is given to the agent, and the second one is used in the a separate async task to detect vad events and trigger some custom audio classification. The one sent to the agent works well, while the second one sometime just does not triggers anything. Any idea?
Copy code
vad = silero.VAD.load(
        min_silence_duration=0.2,
        prefix_padding_duration=0.2,
    )


    voicemail_vad = silero.VAD.load(
        min_silence_duration=0.6,
        prefix_padding_duration=0.2,
    )
t
Thanks for reaching out, aric. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your issue about the second Silero VAD instance (
voicemail_vad
) sometimes not triggering in a separate async task. Short answer: the secondary VAD won’t emit events unless it’s connected to an audio stream and receives frames. The AgentSession’s VAD is auto-wired to room audio, but a standalone VAD must be explicitly streamed and fed audio. What to check / do next: • Ensure you call 
.stream()
on 
voicemail_vad
and subscribe to 
start_of_speech
/ 
end_of_speech
. • Push PCM frames into that stream (e.g., from an 
rtc.AudioStream
of the subscribed track) with 
vad_stream.push_frame(frame)
. • Close the stream when done with 
await vad_stream.aclose()
. • Confirm the audio sample rate is supported (Silero expects 16 kHz/8 kHz; resample if needed). • Add logging inside the VAD event handlers to verify they run and to catch missed frames. If you’d like, paste the async task code that reads the track and how you create/use 
voicemail_vad
and I’ll point out exactly where to add the 
.stream()
/ 
push_frame
logic. Sources: Speech-to-text (STT) models | LiveKit Docs | Silero VAD plugin | LiveKit docs
s
Copy code
@ctx.room.on("track_subscribed")
    def _on_track_subscribed(ev: rtc.RemoteAudioTrack):
        # Find the participant that owns this track for logging
        participant_info = "unknown"
        for participant in ctx.room.remote_participants.values():
            for pub in participant.track_publications.values():
                if pub.sid == ev.sid:
                    participant_info = f"{participant.identity} ({participant.kind})"
                    break
            if participant_info != "unknown":
                break

        <http://logger.info|logger.info>(
            f"Audio track subscribed from participant: {participant_info}, track_sid: {ev.sid}"
        )

        # the function on_audio_track goal's is to capture raw_pcm_chunks in order to classify the audio as human, Voicemail, IVR, or unknown.
        raw_pcm_chunks: list[memoryview[int]] = []
        sample_rate: int = 48000
        num_channels: int = 1

        async def on_audio_track(
            track: rtc.RemoteAudioTrack,
            capture_duration_sec: float = 5.0,  # Capture 5 seconds after speech starts
        ):
            nonlocal \
                call_was_answered, \
                call_answered_at, \
                raw_pcm_chunks, \
                sample_rate, \
                num_channels
            <http://logger.info|logger.info>(
                "Audio track subscribed - starting VAD-based voicemail detection"
            )
            stream = rtc.AudioStream(track)
            <http://logger.info|logger.info>(
                f"voicemail vad identity: {id(voicemail_vad)!s}, pid: {os.getpid()!s}"
            )
            vad_stream = voicemail_vad.stream()

            total_samples_per_ch = 0

            # Initialize with default values to avoid ValueError if no frames are captured

            # VAD state tracking
            speech_started = False
            speech_start_time: float | None = None

            max_samples_per_ch = int(capture_duration_sec * 48_000)

            # Task to monitor VAD events - this stream is separate from session's internal VAD stream
            async def monitor_vad_events():
                nonlocal speech_started, speech_start_time
                <http://logger.info|logger.info>("Monitoring VAD events")
                try:
                    async for event in vad_stream:
                        if (
                            event.type == VADEventType.START_OF_SPEECH
                            and not speech_started
                        ):
                            speech_started = True
                            speech_start_time = event.speech_duration
                            <http://logger.info|logger.info>(
                                f"VAD: Speech started - capturing next {capture_duration_sec}s"
                            )
                        elif event.type == VADEventType.END_OF_SPEECH:
                            <http://logger.info|logger.info>("VAD: Speech ended")

                except asyncio.CancelledError:
                    logger.debug("VAD monitor task cancelled.")
                finally:
                    <http://logger.info|logger.info>("VAD monitoring events finished")

            vad_monitor_task = asyncio.create_task(monitor_vad_events())

            try:
                audio_frame_count = 0
                async for frame in stream:
                    if not call_was_answered:
                        continue

                    f = frame.frame
                    audio_frame_count += 1
                    if audio_frame_count % 100 == 0:
                        <http://logger.info|logger.info>(f"Audio frame count: {audio_frame_count}")

                    # Update sample_rate and num_channels from actual frame data
                    frame_sample_rate = getattr(f, "sample_rate", None)
                    frame_num_channels = getattr(f, "num_channels", None)
                    if frame_sample_rate is not None:
                        sample_rate = frame_sample_rate
                    if frame_num_channels is not None:
                        num_channels = frame_num_channels

                    # Push frames to our VAD stream (separate from session's internal stream)
                    # This stream needs its own input to detect speech
                    vad_stream.push_frame(f)

                    if not speech_started:
                        # Skip storing audio if speech hasn't started yet, but VAD still processes it
                        continue

                    # Store audio for classification (only after speech has started)
                    raw_pcm_chunks.append(f.data)
                    spp = getattr(f, "samples_per_channel", 0)
                    total_samples_per_ch += spp

                    # Stop after capturing the specified duration
                    if total_samples_per_ch >= max_samples_per_ch:
                        <http://logger.info|logger.info>(
                            f"VAD: Captured {total_samples_per_ch / 48000:.2f}s of audio after speech started"
                        )
                        break
            except asyncio.CancelledError:
                logger.debug("Call classification audio track task cancelled.")
            finally:
                # Clean up VAD stream and monitor task
                vad_stream.end_input()
                vad_monitor_task.cancel()
                await vad_monitor_task

            <http://logger.info|logger.info>(
                f"VAD detection complete - Total audio captured: {total_samples_per_ch / 48000:.2f}s"
            )
t
Thanks — I reviewed your code. You’re mostly doing the right things (creating a separate 
voicemail_vad.stream()
, pushing frames, monitoring events, and cleaning up). The intermittent non-triggering is most likely caused by one or more of the issues below. Key action items to try first • Resample audio to 16 kHz before pushing to the VAD. Silero VAD only supports 8000 or 16000 Hz; pushing 48 kHz input will prevent detection. • Ensure the audio is mono. If frames are stereo, downmix to a single channel before 
push_frame
. • Prewarm/load 
voicemail_vad
once (agent startup) rather than per-call to avoid cold-start delays. Small fixes / checks in your code • Don’t rely on 
event.speech_duration
at START_OF_SPEECH; just use 
event.type
(and 
timestamp
if you need timing). • Guard against empty or malformed frames: ◦ skip if 
not hasattr(f, "data") or len(f.data) == 0
• Log frame sample rate and channels right before 
vad_stream.push_frame(f)
to confirm you’re actually sending 16k/mono frames. Suggested minimal resampling sketch • Use an AudioResampler (or your preferred resampling routine) to convert 48000 → 16000 and then push the resampled frames into 
vad_stream
. Cleanup note • 
vad_stream.end_input()
+ canceling the monitor task is fine; ensure 
await vad_monitor_task
runs only after cancel completes (wrap in try/except to avoid propagation of CancelledError). If you want, paste one raw 
frame.frame
object (its attributes: sample_rate, num_channels, samples_per_channel, len(data)) and I’ll show the exact resampling/downmix code to feed the VAD. Sources: Silero VAD plugin | LiveKit docs | VAD Metrics | LiveKit docs | Capturing the First Few Seconds of a Call for Processing
s
I do indeed send 48khz. How do I resample it
t
Good — here’s a concise, working way to resample your 48 kHz frames to 16 kHz and feed them to the VAD. I’ll show two common options: using torchaudio (recommended) and using librosa. 1) Using torchaudio (fast, efficient)
Copy code
import torch
import torchaudio

resampler = torchaudio.transforms.Resample(orig_freq=48000, new_freq=16000)

def frame_to_tensor(frame):
    # frame.data is bytes of int16 PCM
    samples = torch.frombuffer(frame.data, dtype=torch.int16).float()  # [N]
    if frame.num_channels > 1:
        samples = samples.view(-1, frame.num_channels).T  # [channels, samples]
    else:
        samples = samples.unsqueeze(0)  # [1, samples]
    return samples  # float tensor

def tensor_to_frame(tensor, sample_rate=16000):
    # tensor: [channels, samples] or [1, samples]
    if tensor.ndim > 1 and tensor.shape[0] > 1:
        # downmix to mono by averaging channels
        mono = tensor.mean(dim=0)
    else:
        mono = tensor.squeeze(0)
    pcm16 = mono.clamp(-32768, 32767).to(torch.int16).cpu().numpy().tobytes()
    return rtc.AudioFrame(
        data=pcm16,
        sample_rate=sample_rate,
        num_channels=1,
        samples_per_channel=mono.shape[-1],
    )

# Inside your audio loop:
async for frame in stream:
    # Convert frame -> tensor
    samples = frame_to_tensor(frame.frame)          # [channels, samples]
    resampled = resampler(samples)                 # resampler keeps channel dim
    new_frame = tensor_to_frame(resampled, 16000)  # create AudioFrame at 16kHz
    vad_stream.push_frame(new_frame)
Install:
Copy code
pip install torchaudio
2) Using librosa + numpy (simpler, slightly slower)
Copy code
import numpy as np
import librosa

def resample_librosa(frame):
    arr = np.frombuffer(frame.data, dtype=np.int16)
    if frame.num_channels > 1:
        arr = arr.reshape(-1, frame.num_channels).T  # [channels, samples]
        arr = arr.mean(axis=0)                       # downmix to mono
    # librosa expects float32 in [-1,1]
    float_wave = arr.astype(np.float32) / 32768.0
    resampled = librosa.resample(float_wave, orig_sr=frame.sample_rate, target_sr=16000)
    pcm16 = (resampled * 32767).astype(np.int16).tobytes()
    return rtc.AudioFrame(data=pcm16, sample_rate=16000, num_channels=1, samples_per_channel=len(resampled))

# Inside loop:
async for frame in stream:
    new_frame = resample_librosa(frame.frame)
    vad_stream.push_frame(new_frame)
Install:
Copy code
pip install librosa numpy
Notes • Always push 8 kHz or 16 kHz (16 kHz recommended) mono PCM int16 frames to Silero VAD. • Confirm by logging 
frame.frame.sample_rate
and 
frame.frame.num_channels
before and after resampling. • Prewarm/load VAD once if you can, and resample before calling 
vad_stream.push_frame()
. If you want, paste one example 
frame.frame
attributes (sample_rate, num_channels, samples_per_channel, data length) and I’ll provide the exact snippet adapted to it. Sources: Silero VAD plugin | LiveKit docs
Open in Slack
PreviousNext
Powered by