hi

Thanks for reaching out, Aaryan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

so if I use keyboard noise, it works - coz we start and stop keybaord without overlapping with TTS audio track. but for ambient office noise, its a continous stream right, so coz of this I am getting voice breaks in my calls how are we supposed to use ambient noise actually?

Thanks — I’m responding to your question about using continuous ambient office noise and why it’s causing voice breaks. What’s happening • Continuous ambient audio can interfere with speech playout or voice-activity detection if it’s too loud or mixed incorrectly, which causes the breaks you’re seeing. Short sounds like keyboard typing don’t overlap with TTS and so don’t cause the same issue. What to try • Use the BackgroundAudioPlayer’s ambient sound as a separate track and keep its volume lower (example: volume ~0.8) so it doesn’t compete with TTS or trigger VAD. • Start ambient playback after connecting to the room/agent session so the player handles looping cleanly. • If breaks persist, try reducing ambient volume further or using EQ/low-pass to remove frequencies that clash with speech. If you want, share: • which SDK/platform you’re using and a short config snippet (how you start background audio and TTS). I’ll give a targeted config suggestion. Sources: Background audio | LiveKit Documentation, with interactive elements

My AI says:

Thanks — this is a great analysis and I’m with you. You’re right about the root cause: SIP/PSTN needs a single mixed analog line, and the bridge’s mixer is what's stuttering when LiveKit emits multiple concurrent tracks. Given that, the clean path is to avoid relying on the bridge to mix participant tracks. Recommendation • Drop the current PR that relies on the BackgroundAudioPlayer producing a separate ambient track for SIP calls. • Prototype frame-level mixing inside

tts_node

(mix ambience into TTS frames) so the agent sends one outgoing audio track. This avoids SFU/bridge mixing and should eliminate the stutter. Volume tuning and sample-rate alignment will be needed. If you prefer lower effort but accept limitations, you can park the feature for SIP (option: drop feature). Running the built-in BackgroundAudioPlayer on the agent participant will likely still hit the same stutter because the bridge still has to mix. Next steps I can help with • Sketch the frame-level mixing approach and a small implementation plan (mixing loop, ensuring sample-rate/format match, handling silence/TTS boundaries). • Review a prototype diff or PR for

tts_node

(50–200 LOC scope you estimated) and advise on volume/sample handling. Which do you want me to help with first — I can draft the frame-level mixing plan or help close the existing PR and document the SIP limitation. , with interactive elements

• Sketch the frame-level mixing approach and a small implementation plan (mixing loop, ensuring sample-rate/format match, handling silence/TTS boundaries). •

Thanks — I’ll sketch a concise, actionable frame-level mixing plan for

tts_node

covering the mixing loop, format matching, and silence/TTS boundary handling. Overview • Goal: publish a single outgoing audio track by mixing ambient audio into TTS frames inside

tts_node

, avoiding SFU/bridge mixing for SIP/PSTN. • Key constraints: output sample rate, channels, bit depth must match WebRTC/PSTN expectations (e.g., 24000 Hz, mono, PCM_S16LE). Ambient clip should be preprocessed to that format. Implementation sketch 1) Initialization • Load and decode ambient clip once at startup. • Resample/downmix ambient to target format (e.g., 24kHz, mono, 16-bit PCM) and store as a circular buffer of frames matching TTS frame size (samples_per_channel). • Set mixing levels (weights): e.g., TTS=1.0, ambient=0.3 (tunable). 2) Mixing loop (inside

tts_node

) 1. Obtain base TTS frame stream (async generator of AudioFrame). 2. For each TTS frame: ◦ Ensure TTS frame format matches target (resample if needed). ◦ Pull the next ambient frame from circular buffer (advance pointer; loop). ◦ Decide whether to mix: ◦ If TTS frame is non-silent (peak > SILENCE_THRESHOLD), mix ambient into TTS. ◦ If TTS frame is silent, either: ▪︎ Pass through TTS silence (preserve pause), or ▪︎ Optionally send ambient-only frames if you want ambient during silence (but that creates perceived speech during pauses and may reintroduce bridge mixing problems). ◦ Mix at sample level: convert int16 to float32, mixed = clamp(tts_tts_gain + ambient_amb_gain), convert back to int16. ◦ Preserve TTS frame timestamps/metadata; yield mixed frame. 3) Format & resampling • Pre-normalize ambient to target sample rate/channels/bit-depth using a high-quality resampler (libsoxr or equivalent). • If TTS output sample rate differs, resample TTS frames before mixing. • Use PCM_S16LE frames for mixing to match typical RTP payloads; perform math in float32 to avoid clipping, then quantize. 4) Silence detection & boundary handling • Define SILENCE_THRESHOLD as a small amplitude (e.g., max(abs(samples)) < 500 for int16). • For partial-speech frames (low amplitude), still mix but consider dynamic ambient attenuation near speech onsets/offsets (fade-in/out over N frames) to avoid abrupt level jumps. • On TTS end-of-speech: continue ambient loop but stop mixing into new TTS frames; ensure final frames preserve natural tail (apply short fade-out if needed). 5) Buffering, latency, and performance • Prebuffer a few ambient frames to avoid underflow; buffer length depends on frame duration (e.g., 10 frames). • Avoid disk I/O per frame—decode once and keep frames in memory. • Keep mixing CPU-light (vectorized numpy or C extension); mixing per frame is small work if frames are short (10–20 ms). • Ensure thread/async-safety if ambient buffer is fed by a separate task. 6) Edge cases • Sample-rate mismatch between ambient file and runtime TTS: always resample offline when loading ambient if possible. • Stereo ambient: downmix to mono before buffering. • Different frame sizes: either re-chunk ambient frames to match TTS frame_size or use a sample cursor with slicing. Testing checklist • Verify end-to-end on SIP trunk: confirm no stutter and consistent timing. • Test different amb_gain values and fade durations. • Stress test under bridge load to ensure CPU/memory usage is acceptable. • Validate RTT/latency and ensure timestamps on frames align with expected playout. If you want, I can: • Draft a small prototype code snippet for the mixing loop in your code style (Python async + rtc.AudioFrame). • Propose concrete values for frame size, sample rate, SILENCE_THRESHOLD, default gains, and fade lengths to start testing. Which would you prefer? Sources: Background audio | LiveKit Documentation | Pipeline vs. Realtime - Which is the better Voice Agent Architecture? | Liv... | Pipeline nodes and hooks | LiveKit Documentation, with interactive elements

both

Great — below is a concise prototype plus concrete parameter recommendations you can drop into

tts_node

. It mixes ambient audio into TTS frames at the sample level, preserves format/timestamps, handles silence and fades, and is async-friendly. Recommended defaults • sample_rate: 24000 Hz • channels: 1 (mono) • frame_duration: 20 ms → samples_per_frame = sample_rate * 0.02 = 480 samples • TTS gain: 1.0 • ambient gain: 0.3 (tune 0.1–0.5) • SILENCE_THRESHOLD: 500 (int16 peak) • FADE_FRAMES: 3 (for 60 ms fade-in/out with 20 ms frames) • ambient buffer: preload 10 frames Prototype (Python async, uses numpy; adapt to your AudioFrame type)

import asyncio
import numpy as np

SAMPLE_RATE = 24000
CHANNELS = 1
FRAME_SAMPLES = int(SAMPLE_RATE * 0.02)  # 20 ms
TTS_GAIN = 1.0
AMB_GAIN = 0.3
SILENCE_THRESHOLD = 500
FADE_FRAMES = 3
AMBIENT_BUFFER_SIZE = 10

class AmbientMixer:
    def __init__(self, ambient_frames_bytes_list):
        # ambient_frames_bytes_list: list of raw PCM int16 frames already resampled/downmixed
        self.frames = ambient_frames_bytes_list
        self.idx = 0
        self.lock = asyncio.Lock()

    async def next_frame(self):
        async with self.lock:
            b = self.frames[self.idx]
            self.idx = (self.idx + 1) % len(self.frames)
        return b

def int16_to_float32(buf):
    return np.frombuffer(buf, dtype=np.int16).astype(np.float32)

def float32_to_int16(arr):
    arr = np.clip(arr, -32768, 32767)
    return arr.astype(np.int16).tobytes()

def is_silent_int16(buf):
    arr = np.frombuffer(buf, dtype=np.int16)
    return np.max(np.abs(arr)) < SILENCE_THRESHOLD

def apply_fade(arr, fade_in=False, fade_out=False, frames=FADE_FRAMES):
    # arr: float32 array shaped (N,)
    # apply linear fade over frames*FRAME_SAMPLES samples
    length = arr.shape[0]
    fade_len = min(frames * FRAME_SAMPLES, length)
    if fade_in:
        ramp = np.linspace(0.0, 1.0, fade_len, endpoint=True)
        arr[:fade_len] *= ramp
    if fade_out:
        ramp = np.linspace(1.0, 0.0, fade_len, endpoint=True)
        arr[-fade_len:] *= ramp
    return arr

async def tts_node_mixed(base_tts_stream_async_iterable, ambient_mixer: AmbientMixer):
    """
    base_tts_stream_async_iterable yields objects with:
      - .data: raw PCM int16 bytes for FRAME_SAMPLES samples (mono)
      - .sample_rate, .num_channels, .samples_per_channel, .timestamp (if present)
    Yields same AudioFrame objects with mixed .data.
    """
    prev_tts_was_speech = False

    async for tts_frame in base_tts_stream_async_iterable:
        # ensure TTS frame format matches; resample outside this loop if needed
        tts_bytes = tts_frame.data
        # pull ambient
        ambient_bytes = await ambient_mixer.next_frame()

        # Silence detection on TTS frame
        tts_is_silent = is_silent_int16(tts_bytes)

        if not tts_is_silent:
            # mix
            tts_f = int16_to_float32(tts_bytes) * TTS_GAIN
            amb_f = int16_to_float32(ambient_bytes) * AMB_GAIN

            # optional fades at boundaries
            if not prev_tts_was_speech:
                # fade ambient in a bit (reduce abruptness at speech start)
                amb_f = apply_fade(amb_f, fade_in=True, fade_out=False)
            # mix and clamp
            mixed = tts_f + amb_f
            mixed_bytes = float32_to_int16(mixed)
            prev_tts_was_speech = True
        else:
            # preserve silence (do not overlay ambient) to keep intended pauses
            mixed_bytes = tts_bytes
            # optionally, if you want ambient during pure silence (careful for SIP):
            # mixed_bytes = float32_to_int16(int16_to_float32(ambient_bytes) * AMB_GAIN)
            prev_tts_was_speech = False

        # construct and yield a frame preserving metadata (adapt to your AudioFrame class)
        out = type(tts_frame)(data=mixed_bytes,
                              sample_rate=tts_frame.sample_rate,
                              num_channels=tts_frame.num_channels,
                              samples_per_channel=tts_frame.samples_per_channel)
        yield out

    # when base stream ends, nothing special—ambient continues in other contexts as needed

Notes and integration tips • Preprocess ambient file once: decode, resample/downmix to SAMPLE_RATE/mono/PCM_S16LE, and slice into FRAME_SAMPLES chunks stored as bytes list for AmbientMixer. • If TTS output sample rate can vary, resample TTS frames to SAMPLE_RATE before feeding the mixing loop (use soxr or a fast resampler). • Use numpy vector ops for speed; if perf becomes an issue, implement mixing in native code. • Tune AMB_GAIN and SILENCE_THRESHOLD with real SIP call tests. If you see residual artifacts at phrase boundaries, increase FADE_FRAMES to smooth transitions. • If you want ambient present during TTS silence (e.g., continuous background in hold music), enable the commented ambient-on-silence branch—note this may reintroduce mixing load on bridges if you then publish separate tracks elsewhere. If you want, I can: • Draft the ambient preprocessor (resample/downmix + chunking) next, or • Convert this snippet to match your exact AudioFrame class and integrate into your

tts_node

function. Which should I do first?, with interactive elements