background_audio issue

Thanks for reaching out, Aaryan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

# BackgroundAudioPlayer breaks SIP inbound audio (STT dies) ## Environment - livekit-agents SDK (Python) - SIP telephony via Exotel connector - LiveKit Cloud ## Problem When using

BackgroundAudioPlayer

during a SIP call, the inbound audio stream from the caller stops working. STT receives no more transcripts after the background audio track is published. The SIP bridge appears to stop forwarding the caller's microphone audio. ## Timeline from logs

13:23:33.603 - STT FINAL received: "हां बोलिए" (last successful STT)
13:23:34.299 - BackgroundAudioPlayer created + started (publish_track)
13:23:34.299 - Thinking audio starts playing (keyboard typing clip)
13:23:35.356 - Thinking audio stopped, TTS response plays normally
              ... user speaks but ZERO STT events after this point ...
13:24:07.410 - "stream closed" for participant microphone (SOURCE_MICROPHONE)
13:24:18.555 - Inactivity fires: stt_idle=45.0s (no STT since 13:23:33)

## What we're doing We play a "keyboard typing" sound while the LLM is thinking, before TTS responds. We use

BackgroundAudioPlayer

for this:

python
from livekit.agents import BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip

# Create player and publish track
player = BackgroundAudioPlayer()
await player.start(room=room, agent_session=session)

# Play keyboard typing sound in a loop
clip = BuiltinAudioClip.KEYBOARD_TYPING
handle = player.play(AudioConfig(clip, volume=0.8), loop=True)

# Later, stop it before TTS speaks
handle.stop()
player._audio_source.clear_queue()

## Root cause

BackgroundAudioPlayer.start()

calls

publish_track()

which publishes a second audio track (

background_audio

) to the room. The SIP bridge seems unable to handle multiple audio tracks from the agent participant — after the second track appears, it stops forwarding the caller's inbound microphone audio to STT. ## What we tried 1.

agent_session=None

— same issue, plus call drops when track is unpublished 2.

agent_session=session

— STT dies after publish_track 3. Creating player once and reusing (no repeated publish/unpublish) — STT still dies after first publish_track 4. Avoiding

aclose()

(no unpublish_track) — STT still dies because the publish itself is the problem 5. Writing directly to

session.output.audio.capture_frame()

— blocks TTS from playing (they fight over the same output) ## Question Is there a way to play background/thinking audio during a SIP call without publishing a separate audio track? Ideally we'd mix it into the agent's existing audio track. Or is there a SIP bridge configuration to handle multiple audio tracks from the same participant?

Thanks — this is very clear and the logs are helpful. I’m responding to your question about whether you can play background/thinking audio during a SIP call without publishing a separate audio track (or otherwise configure the SIP bridge to accept multiple tracks). Short answer • The SIP bridge does not support multiple concurrent audio tracks from the same participant. Publishing the BackgroundAudioPlayer’s second audio track causes the SIP bridge to stop forwarding the caller’s inbound microphone audio, which stops STT. • There’s no supported SIP-bridge configuration to accept/merge multiple agent tracks from a single participant. • Mixing the background clip into the agent’s existing audio track at the transport level is not provided as a built-in feature, and writing directly to

session.output.audio.capture_frame()

conflicts with TTS, so that approach isn’t viable. What this means for you • Using

BackgroundAudioPlayer.start()

(which publishes a second

background_audio

track) will continue to break inbound SIP audio/STT in the current environment. • Avoiding publish/unpublish or reusing the same player won’t help because the issue happens when the second track is published. Workarounds you can try • Mix the background sound into the agent’s output before it reaches LiveKit (client-side mixing) so only a single agent track is published: a. Create a single audio output stream that combines (a) the agent’s TTS and (b) the background/typing clip locally in your process. b. Publish only that single mixed stream to LiveKit so the SIP bridge still sees one audio source. ◦ Note: you mentioned

session.output.audio.capture_frame()

blocked TTS when used; that indicates you’ll need a mixing pipeline that combines TTS and background audio frames together before writing them to the single capture API rather than trying to write separate streams concurrently. • Perform server-side mixing (if feasible in your architecture): ◦ Render/mix the background clip with TTS on your server process into one PCM stream, then push that single stream into the LiveKit agent output. • Avoid background audio during active inbound speech: ◦ Only play the background clip locally to your agent operator (if there is a human operator) or skip it when an inbound caller is speaking so STT continuity is preserved. This is a mitigation, not a full solution. Questions to help narrow the best approach 1. Is the agent audio (TTS) generated and sent from the same process that can perform PCM-level mixing? Or are TTS and background playback handled by separate services/processes? 2. Do you have control over the capture pipeline used to write audio into LiveKit (i.e., can you mix frames before calling

capture_frame()

), or must you publish via LiveKit-provided track APIs only? 3. Is there any requirement that the background sound be a separate track (e.g., for independent volume control by consumers)? If yes, we’ll need to consider alternative architectures (for example, mixing locally but exposing control metadata to clients). If you’d like, I can: • Walk through an example mixing flow for combining a background clip + TTS into a single PCM stream and pushing that via

session.output.audio.capture_frame()

without blocking TTS (I’ll need details about how your TTS frames are currently produced and how you’re calling

capture_frame()

). • Help design a server-side mixer outline if TTS runs on the server. Thanks for the clear reproduction steps and logs — once you tell me which mixing options are possible in your setup (answers to Q1–Q3), I’ll provide a concrete implementation path you can try. Sources: Agent speech and audio | LiveKit Documentation | sip | livekit/livekit_composite | DeepWiki

so then how you guys work with background_audio without such issues?

Thanks — I hear your frustration. You’re asking how LiveKit handles background audio without breaking SIP inbound audio/STT, given your reproduction where publishing BackgroundAudioPlayer’s second track causes the SIP bridge to stop forwarding caller audio. Short answer • LiveKit’s BackgroundAudioPlayer publishes a separate background audio track. In architectures where the SIP bridge accepts multiple tracks per participant, that works fine; in your setup the SIP bridge stops forwarding inbound caller audio when a second agent track appears, which is why STT stops. • Because the SIP bridge you’re using does not support multiple concurrent audio tracks from the same participant, the safest approach is to ensure only a single mixed audio track is published for the agent when bridging to SIP. How LiveKit avoids breakage in environments that work • Background audio is published on a dedicated track separate from TTS/voice so it can be looped or controlled independently without altering the agent’s primary voice track. • In environments where the SIP bridge and media path support multiple outgoing tracks from one participant and correctly multiplex/forward them, inbound audio and STT remain unaffected because incoming and outgoing paths are isolated. • Noise cancellation and other processing are applied per participant/stream so incoming SIP audio is handled independently from agent output. Why you’re seeing STT stop • Publishing a second agent-originated audio track changes the participant’s outbound track set. Your SIP bridge treats that change as a signal to reconfigure or stop forwarding the caller’s inbound audio, so the STT pipeline sees no more remote audio. This is a bridge-level media-routing limitation rather than an STT bug. Practical options for you • Mix locally before publishing (recommended) a. Combine background/typing audio + TTS into a single PCM stream in your process. b. Publish that single mixed stream to LiveKit (or write it once via

session.output.audio.capture_frame()

), so the SIP bridge sees only one agent audio source. ◦ Important: don’t attempt to write TTS and background audio as separate concurrent writes; instead produce a single mixed frame and send that. • Server-side mixing ◦ Render/mix TTS + background clip on the server into one stream, then send that stream to LiveKit. • Conditional background playback (mitigation) ◦ Only play the background clip when the caller is not speaking (VAD) or play it locally for your operator, not into the published track. • If you must have separate control over background volume for listeners, send control signals/metadata to clients so they can locally mix or adjust playback (but still publish only one mixed stream to the SIP bridge). Questions to recommend a concrete implementation 1. Is your TTS generation and publishing happening in the same process that can perform PCM-level mixing? 2. How are you currently using

session.output.audio.capture_frame()

— are you able to intercept or mix frames before calling it? 3. Do you need independent volume control for background audio for downstream listeners, or is a single mixed stream acceptable? If you can tell me which of the above fits your setup (answers to Q1–Q3), I’ll provide a concrete mixing example and step-by-step changes you can implement to avoid publishing a second track while keeping your thinking-sound behavior. Sources: Agent speech and audio | LiveKit Documentation | livekit/livekit_composite | DeepWiki | External data and RAG | LiveKit Documentation | Choosing the right noise cancellation for your agent | LiveKit

can we add a second participant then to the same room to fix this?

Thanks — good question. I’m responding to your ask about adding a second participant to avoid the SIP-bridge single-track limitation. Short answer • Yes — adding a second participant into the same room is a viable workaround. Put the SIP caller and STT pipeline on one participant, and publish background/ambient audio from a separate participant. This prevents the SIP bridge from seeing multiple outgoing tracks from the SIP participant and avoids breaking inbound audio/STT. How this helps (brief) • The SIP bridge treats each participant separately. If the agent’s background audio is published as a distinct participant, the SIP participant’s inbound audio routing and STT remain intact. Each participant can publish their own tracks without the bridge misrouting or dropping the caller audio. Practical approaches • Option A — Create a lightweight “background” participant a. Create a second participant (can be an automated bot/agent identity) that joins the same room. b. Use that participant to publish only the background/ambient track(s). c. Keep your SIP participant as-is so STT continues to receive the caller’s inbound audio. ◦ Pros: Minimal changes to SIP lifecycle, simple separation of concerns. ◦ Cons: Slightly more bandwidth and one extra participant to manage. • Option B — Agent-as-two-participants (one for voice, one for ambience) ◦ If you need the agent’s TTS and background audio to be controlled independently, run two agent instances: one publishes TTS (and is used for any server-side audio processing), the other publishes background audio only. • Option C — Client- or server-side mix + single participant (preferred if you want no extra participants) ◦ If you can mix background + TTS into a single stream, publish that single mixed stream from one participant. This avoids the SIP bridge issue but requires building a mixing pipeline. Implementation notes & gotchas • Participant identity and audio routing: ◦ Make sure the STT and any logic that consumes inbound audio uses the SIP participant’s RemoteAudioTrack (not the background participant). ◦ If you have session logic that automatically selects “the agent” by identity, ensure it targets the correct participant (the one carrying the caller audio). • Bandwidth and SFU: ◦ Adding a participant uses extra upstream bandwidth, but LiveKit’s SFU handles multiple participants efficiently. • Avoid self-processing: ◦ Configure your processing pipeline so the background participant’s audio is not mistakenly fed into your STT (i.e., ignore local/background participant tracks when selecting STT input). • Synchronization: ◦ If you need tight sync between TTS and background audio (e.g., fade-out when TTS starts), coordinate across the two agent processes or send control events between participants so the background participant stops/fades when TTS begins. Questions to tailor a step-by-step guide 1. Do you control the SIP participant lifecycle separately from the agent process, or is everything currently handled by a single agent instance? 2. Is low-latency sync required between TTS start and background fade-out? 3. Do you need independent volume control for background audio for downstream listeners (or is a simple on/off enough)? If you tell me which option you prefer and answer the questions above, I’ll provide a concrete example (code snippet and lifecycle steps) showing how to: • create the second participant, • publish background audio from it, • ensure STT continues to consume only the SIP participant’s audio, and • coordinate fades/stops between participants if you need synchronization. Sources: SIP Lifecycle Management Agent | LiveKit Documentation | Outbound Calling Script | LiveKit Documentation | livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki | Agent speech and audio | LiveKit Documentation