https://livekit.io logo
Join Slack
Powered by
i feel that when my agent is interrupted, the TTS ...
# ask-ai
a
i feel that when my agent is interrupted, the TTS is cut how to make it smoother
t
Thanks for reaching out, Tom. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I hear you: you want the agent’s TTS to stop more smoothly instead of cutting off abruptly when interrupted. I can help you with ways to make interruptions sound natural. Quick summary of what’s happening • By default, TTS stops immediately when interrupted, which causes an abrupt cutoff. You can make this smoother by fading the audio out before stopping. Two practical approaches (pick one based on how much control you have over audio output) • Option A — Implement a fade-out in your audio output node (recommended when you can customize playback) a. Detect interruption by checking the agent’s 
SpeechHandle.interrupted
flag. b. When interruption is detected, apply a short volume ramp-down (fade) across a few audio frames (for example 8–20 frames, depending on your frame size and desired fade length). c. Stop sending audio after the fade completes and clear the current speech handle. What this gives you: a natural-sounding tail rather than an abrupt cut. This is the solution when you can extend 
tts_node
or 
realtime_audio_output_node
and process audio frames before they are played. • Option B — Design-level changes if you can’t modify audio output ◦ Use shorter utterances so there’s less chance of cutting mid-sentence. ◦ Insert brief pauses or explicit turn-taking points between sentences so the agent checks for input before continuing. ◦ Break long replies into multiple smaller TTS calls so interruptions occur at natural breakpoints. If you want to implement the fade-out, I can provide a concise code example tailored to your stack. Helpful details to tell me so I give the right snippet: • Which SDK/language are you using (Python, Node, or another)? • Are you able to modify the 
tts_node
or the audio playback pipeline directly? • If applicable, what sample rate and frame size do you use, or do you use raw PCM frames? Tell me those and I’ll give a targeted implementation example (short, runnable snippet) showing how to detect interruption and apply a smooth volume ramp-down. Sources: How to Handle Speech Interruptions More Naturally | Agent speech and audio | LiveKit Documentation
a
please show me the fade out
t
Thanks — I’ll show a fade-out example, and to make it as useful as possible I need to know which language/SDK you’re using. Could you tell me: • Which SDK/language: Python or Node.js (or another)? • Are you able to modify the 
tts_node
or 
realtime_audio_output_node
directly, or do you only control the audio frames before playback? • If you know it: the audio sample rate (e.g., 48000 Hz) and whether frames are 20ms, 10ms, or variable. If you want Python, I’ll provide a short runnable snippet that: 1. Detects 
SpeechHandle.interrupted
. 2. Applies a gradual volume ramp-down across a few frames. 3. Stops playback after the fade completes. If you want Node.js, I’ll provide an equivalent example using audio buffers. Which one would you like?
a
i am in python
t
Thanks — I’ll give you a concise, practical Python example that shows how to apply a smooth fade-out when TTS is interrupted. This example assumes you can override the agent’s 
tts_node
(or otherwise intercept audio frames prior to playback) and that audio frames are 16-bit PCM. What this does • Detects when 
context.speech_handle.interrupted
becomes True. • Applies a linear fade-out over a configurable duration (default 200 ms) to the remaining audio frames. • Stops playback after the fade completes. Example: fade-out in Python
Copy code
import itertools
import numpy as np
from typing import AsyncIterable
from livekit import rtc
from livekit.agents import Agent, ModelSettings, RunContext

class SmoothInterruptAgent(Agent):
    def __init__(self, fade_ms: int = 200) -> None:
        super().__init__(instructions="You are a helpful agent with smooth interruption handling.")
        self.fade_ms = fade_ms  # fade duration in milliseconds

    async def tts_node(
        self,
        text: AsyncIterable[str],
        model_settings: ModelSettings,
        context: RunContext,
    ) -> AsyncIterable[rtc.AudioFrame]:
        # Use default TTS generation then post-process frames for fade-out
        async for frame in Agent.default.tts_node(self, text, model_settings):
            # If not interrupted, pass through
            if not context.speech_handle.interrupted:
                yield frame
                continue

            # Interrupted: perform fade-out on this and subsequent frames until fade completes
            sample_rate = frame.sample_rate
            channels = frame.num_channels
            samples_per_channel = frame.samples_per_channel

            # total samples to fade (per channel)
            fade_samples = int((self.fade_ms / 1000.0) * sample_rate)
            if fade_samples <= 0:
                # Nothing to do; stop immediately
                break

            # Convert frame bytes (16-bit little-endian) to numpy array
            pcm = np.frombuffer(frame.data, dtype=np.int16)

            # We may need to supply several frames worth of audio to complete fade.
            # Start by building a stream of PCM from current frame and subsequent frames.
            # Create an iterator starting with the current frame, then the generator of remaining frames.
            async def remaining_pcm_iter():
                # yield current frame first
                yield pcm
                async for next_frame in Agent.default.tts_node(self, text, model_settings):
                    # stop yielding if no more frames
                    yield np.frombuffer(next_frame.data, dtype=np.int16)

            # Flatten frames until we have at least fade_samples * channels samples
            collected = []
            total_needed = fade_samples * channels
            collected_count = 0
            async for chunk in remaining_pcm_iter():
                collected.append(chunk)
                collected_count += chunk.size
                if collected_count >= total_needed:
                    break

            if collected_count == 0:
                break

            all_pcm = np.concatenate(collected)

            # Generate a fade multiplier: linear from 1.0 down to 0.0 across fade_samples (per channel)
            # We must create a multiplier for interleaved channels: repeat per channel
            fade = np.linspace(1.0, 0.0, fade_samples, endpoint=True, dtype=np.float32)
            fade_interleaved = np.repeat(fade, channels)

            # If all_pcm is longer than fade length, truncate after fade; otherwise use available length
            effective_len = min(all_pcm.size, fade_interleaved.size)
            faded = all_pcm[:effective_len].astype(np.float32) * fade_interleaved[:effective_len]

            # Convert back to int16
            faded_int16 = np.clip(faded, -32768, 32767).astype(np.int16)

            # Yield one (or more) frames made from faded data, split to match original frame sizes
            # Use the original frame's samples_per_channel to split into frames
            frame_sample_count = samples_per_channel * channels
            start = 0
            while start < faded_int16.size:
                end = start + frame_sample_count
                chunk = faded_int16[start:end]
                # If chunk is shorter than full frame, pad with zeros
                if chunk.size < frame_sample_count:
                    pad = np.zeros(frame_sample_count - chunk.size, dtype=np.int16)
                    chunk = np.concatenate([chunk, pad])
                yield rtc.AudioFrame(
                    data=chunk.tobytes(),
                    sample_rate=sample_rate,
                    num_channels=channels,
                    samples_per_channel=samples_per_channel,
                )
                start = end

            # Fade complete — stop sending further speech
            break
Notes and tuning • Fade length: change 
fade_ms
to taste (e.g., 100–300 ms is usually natural). Longer fades may sound more graceful but risk overlapping user speech. • Frame sizes: the snippet preserves the original 
samples_per_channel
per yielded AudioFrame. If your frames are variable length, adapt splitting accordingly. • Channels: code assumes interleaved 16-bit PCM (common for many TTS outputs). If your TTS uses float PCM or different bit depth, convert accordingly. • Efficiency: this example accumulates just enough samples to complete the fade. If your pipeline yields many small frames, you may simplify by applying a fade directly inside each frame using a running fade counter rather than concatenating frames. If you want, I can: • Provide a variant that applies the fade in-place per-frame as soon as the interruption is detected (no concatenation). • Adjust the snippet to match your exact sample rate and frame duration (tell me sample rate and ms per frame). • Convert this to a Node.js example if needed later. Which follow-up would you prefer — an in-place per-frame fade (simpler, lower memory) or the exact sample-rate/frame-size adaptation for your environment? Sources: How to Handle Speech Interruptions More Naturally | Pipeline nodes and hooks | LiveKit Documentation | Agent speech and audio | LiveKit Documentation
2 Views
Open in Slack
PreviousNext
Powered by