https://livekit.io logo
Join Slack
Powered by
We're running a self-hosted LiveKit setup (livekit...
# ask-ai
b
We're running a self-hosted LiveKit setup (livekit-server, livekit-sip, livekit-egress) for an inbound voice AI agent. Calls come in via Twilio Elastic SIP Trunk → LiveKit SIP → Agent (Python SDK). The agent uses ElevenLabs TTS (eleven_turbo_v2_5) and Deepgram STT (flux-general-en). The audio quality sounds noticeably grainier/less crisp compared to the same Twilio trunk used with VAPI. Both are negotiating PCMU/8000, so the codec is the same. Our current config: - audio_output sample_rate: 16000 (downsampled to 8kHz for G.711) - noise_cancellation: BVCTelephony() - turn_detection: "stt" with Silero VAD - ElevenLabs voice settings: stability=0.5, similarity_boost=0.3, speed=1.15 Questions: 1. Are there any recommended audio output settings (sample rate, buffer size, etc.) for self-hosted SIP deployments to maximize voice quality over G.711? 2. Does the LiveKit SIP service do any audio processing (gain, compression, resampling) that we can tune? 3. Is G.722 negotiation possible with Twilio via the LiveKit SIP service, or is it limited to G.711? 4. Any other tips for getting the best audio quality on a self-hosted stack?
✅ 1
t
Thanks for reaching out, Randy. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a clear description and very helpful context. I’ll respond to each of your four questions and give practical items you can try to isolate and improve the perceived “graininess.” Acknowledgement • You described a self‑hosted stack (livekit-server, livekit-sip, livekit‑egress) with Twilio Elastic SIP Trunk → LiveKit SIP → Python agent. Both paths negotiate PCMU/8000 and you synthesize at 16 kHz then downsample to 8 kHz. You’re using BVCTelephony() noise cancellation, Silero VAD, ElevenLabs 
eleven_turbo_v2_5
with stability=0.5 / similarity_boost=0.3 / speed=1.15, and Deepgram STT. 1) Recommended audio output settings for G.711 (PCMU) • Source sample rate: Keep your TTS output at 16 kHz if possible; that gives more headroom before downsampling to 8 kHz and avoids generating extra quantization/noise at synthesis time. • Output format: Use raw PCM (16‑bit linear PCM) from the TTS engine before handing audio to LiveKit so you avoid lossy container/codec artifacts prior to G.711 encoding. • Downsampling: Make sure downsampling to 8 kHz is performed with a good resampler (linear interpolation or better). Poor resampling algorithms can introduce graininess. • Buffering/frames: Produce steady frames (e.g., 20–30 ms frames) rather than many tiny fragments; smaller packets can increase processing overhead and perceived artifacts. While LiveKit doesn’t expose a single “buffer size” knob for SIP trunks, ensure your audio pipeline produces consistent frame sizes. • Loudness / headroom: Ensure your TTS output has reasonable headroom (no heavy clipping/peaking) before G.711 conversion. G.711 has limited dynamic range and clipping will sound harsh. 2) Does LiveKit SIP perform audio processing you can tune? • LiveKit SIP does perform server‑side audio handling (transcoding/resampling and optional server side noise cancellation). That processing can change the audio character. • There is a server trench of operations that can include resampling and gain/processing when bridging SIP and WebRTC. • You can disable server‑side noise cancellation (e.g., 
krisp_enabled
) on the SIP trunk. If you keep agent-side BVCTelephony(), disable server NC to avoid double processing — double noise cancellation often causes speech to sound muffled or “gritty.” • There aren’t exposed fine‑grained knobs for per‑call gain/compression in the SIP gateway; the main actionable control is enabling/disabling server-side NC and ensuring your agent supplies clean PCM input. 3) Is G.722 possible via Twilio + LiveKit SIP? • G.722 negotiation is not supported through LiveKit SIP. The LiveKit SIP gateway negotiates narrowband G.711 (PCMU/PCMA) for SIP trunks. You cannot rely on G.722 via the LiveKit SIP integration currently. • Because WebRTC endpoints support wideband codecs (Opus) inside LiveKit, the limitation is at the SIP trunk gateway boundary — the SIP side will be limited to G.711. 4) Other practical tips to maximize audio quality on a self‑hosted stack • Avoid cascading noise reduction: Only one layer of NC (agent-side BVCTelephony() OR SIP trunk krisp) should be active. • Record and compare raw streams: ◦ Capture raw RTP/PCM directly from Twilio before it enters LiveKit to see whether degradation is happening on the Twilio→LiveKit hop or inside LiveKit/agent processing. ◦ Capture the audio your agent receives (pre- and post-processing) and the audio sent to Twilio. Compare waveforms and spectra to find where high‑frequency loss or distortion appears. • Verify resampling quality: Ensure the resampler used when going from 16 kHz → 8 kHz is high quality. If your agent code performs resampling, try using a well‑tested resampling library. • VAD tuning: Tweak Silero VAD thresholds (activation threshold, min silence length). Aggressive VAD can clip speech on start/stop which may be perceived as lower quality. • TTS output settings: You already have reasonable ElevenLabs settings. Also: ◦ Request PCM output from ElevenLabs (avoid container/compressed outputs). ◦ Try slightly lower stability or similarity_boost in listening tests — depending on the voice, too much processing can remove natural transients that make speech sound “crisp.” ◦ Test speed = 1.0 temporarily to rule out time‑scale artifacts at higher speeds. • System resources and latency: ◦ Ensure your self‑hosted servers have sufficient CPU and low latency between livekit-server, sip, and the agent. CPU contention or packet buffering can introduce artifacts. ◦ Avoid packet fragmentation and excessive jitter on the host network. • STT chain impact: Noise suppression and resampling impact both human listenability and STT accuracy. When optimizing for human listening, check STT results as well; sometimes improving STT alignment (less aggressive NC) also improves perceived clarity. • If possible, test a direct comparison: connect Twilio trunk directly to a system that plays your ElevenLabs audio (bypassing LiveKit). This isolates whether LiveKit processing is the root cause. Short checklist to try next (practical steps) 1. Disable 
krisp_enabled
(server-side NC) on the SIP trunk so only BVCTelephony() runs. Test audio again. 2. Confirm ElevenLabs is returning 16‑bit PCM and that downsampling to 8 kHz uses a good resampler. 3. Temporarily set ElevenLabs speed to 1.0 and slightly lower `stability`/`similarity_boost` for an A/B test. 4. Capture raw audio at three points: Twilio ingress, LiveKit SIP output to agent, and agent send to Twilio. Compare. 5. Tune Silero VAD thresholds to ensure speech boundaries aren’t clipped. If you want, I can help you more specifically with: • Exact places/config keys to toggle 
krisp_enabled
in your SIP trunk config. • A short list of resampler libraries and sample code to downsample from 16 kHz → 8 kHz with minimal artifacts. • Guidance on how to capture and compare the three raw streams mentioned above so we can pinpoint where the graininess appears. Would you like me to walk through any of those next steps (pick one), or share the SIP trunk config and an example agent audio pipeline so I can point to exact settings? Sources: Build a healthcare intake assistant with an Anam avatar | LiveKit | Audio customization | LiveKit Documentation | SIP Telephony and Encryption | Agents telephony integration | LiveKit Documentation | Choosing the right noise cancellation for your agent | LiveKit, with interactive elements
b
Is G.722 possible?
t
Short answer: No — G.722 is not supported through the LiveKit SIP gateway. LiveKit’s SIP integration negotiates narrowband G.711 (PCMU/PCMA) for SIP trunks, so you cannot use G.722 via Twilio → LiveKit SIP. If you need wideband audio, two alternatives I can help you evaluate: • Keep Opus/G.722‑like wideband audio between WebRTC endpoints inside LiveKit and only accept narrowband at the SIP boundary (improve quality on the WebRTC side). • Bypass the LiveKit SIP gateway for the SIP trunk (if possible) or use a different gateway that supports G.722. Would you like help exploring either option or testing where the quality drop occurs? , with interactive elements
b
Follow-up: Audio still choppy after isolating NC Thanks for the tips. I tested with server-side krisp only (no agent-side NC) and the audio is still choppy — sounds like brief dropouts or jitter artifacts rather than just low-fidelity narrowband. The same SIP trunk sounds clean when used with a different voice AI platform, so the issue seems specific to the LiveKit processing/transport path. A few things I noticed in our logs: - jitterBuf: false on all SIP calls - The agent worker is not colocated with the LiveKit server (connecting over the internet) Questions: 1. Should the SIP jitter buffer be enabled? What does jitterBuf: false mean — is it a config we control or determined automatically? 2. Could network latency between the agent worker and the LiveKit server cause choppy audio? Would colocating them on the same network help? 3. Are there any livekit-server or livekit-sip config options for improving audio smoothness (packet loss handling, buffer sizes, etc.)? 4. Any known issues with streaming TTS providers causing choppy playback through LiveKit?
t
Thanks — this is a great follow-up and the log details help narrow things down. I’ll address each question and give concrete next steps to diagnose and fix the choppy/dropout behavior. Acknowledgement • You verified NC isolation (server-side krisp only) and still hear choppy audio (dropouts/jitter). You noted logs show 
jitterBuf: false
and that the agent worker is not colocated with the LiveKit server. 1) Should the SIP jitter buffer be enabled? What does 
jitterBuf: false
mean? • 
jitterBuf: false
in your logs indicates the SIP jitter buffering behavior is not active on that media path. In practice, LiveKit’s media pipeline uses adaptive buffering/congestion control for WebRTC, but the SIP gateway path may bypass or not enable an additional jitter buffer in some situations. • You do not normally toggle a per-call “jitterBuf” flag in everyday configs — LiveKit manages buffering automatically. However, if the SIP path is showing 
false
, it signals that the SIP→WebRTC bridge is not adding extra buffering to absorb network jitter, which can produce the dropouts you’re hearing. • Practical next step: Capture SIP call logs and RTP metrics (packet loss, jitter, reordering) around a failing call. Those metrics will confirm whether missing buffering or network jitter is the root cause. 2) Could network latency between agent worker and LiveKit server cause choppy audio? Would colocating help? • Yes. Network latency, jitter, and packet loss between the agent worker and the LiveKit server can directly cause choppiness and brief dropouts. Every extra network hop increases the chance of packet delay/arrival variability. • Colocating the agent worker with the LiveKit server (same LAN or same cloud region / VPC) will reduce RTT and jitter and is very likely to improve smoothness. This is one of the highest-impact mitigations you can try quickly. 3) Are there livekit-server / livekit-sip config options for improving audio smoothness? • There are no widely exposed knobs for manual jitter buffer size or packet-loss concealment in standard self-hosted configs. LiveKit performs automatic congestion control and jitter management for WebRTC streams, but the SIP bridging layer’s buffering behavior is less configurable. • Things you can control that help: ◦ Agent-side buffering: Increase the agent’s audio source buffer to smooth input (see next point on 
queue_size_ms
). ◦ AudioSource queue: Increase 
queue_size_ms
in the agent SDK (example below) to pre-buffer more TTS audio and prevent underruns. ◦ Network-level tuning: Ensure low-latency networking between components (colocation, QoS, MTU, firewall rules allowing RTP in both directions). ◦ Resource provisioning: Ensure the livekit-server and livekit-sip processes have CPU headroom — heavy CPU load can cause packet handling delays. • Example (agent-side) — increase buffer:
Copy code
# example pattern for AudioSource in agent SDK
  audio_source = rtc.AudioSource(sample_rate=24000, num_channels=1, queue_size_ms=2000)
A larger 
queue_size_ms
(e.g., 1000–2000 ms) helps absorb intermittent delays in TTS chunk delivery or network jitter. 4) Any known issues with streaming TTS providers causing choppy playback through LiveKit? • Streaming TTS can cause choppy playback if audio chunks arrive irregularly or the agent publishes audio in small bursts. Even with perfect network, small gaps in feeding audio frames => perceived dropouts. • Common causes: ◦ TTS chunking or rate-limiting (provider sends data in uneven bursts). ◦ Agent code not buffering or concatenating TTS frames into steady 20–30 ms frames before publishing. ◦ Resampling or transcoding delays at the agent when converting sample rates (16 kHz → 8 kHz). • Mitigations: ◦ Request raw PCM (16-bit) from the TTS provider and ensure continuous streaming flow to your agent. ◦ Buffer TTS output on the agent (use 
queue_size_ms
) and publish steady frames to LiveKit. ◦ If your agent resamples, use a high-quality resampler and do resampling before enqueueing into the audio source buffer. ◦ Temporarily test with a pre-recorded continuous PCM file (played at source) to verify whether LiveKit path handles continuous audio smoothly. If pre-recorded audio plays cleanly while streamed TTS is choppy, the TTS delivery is likely the culprit. Concrete diagnostic checklist (order to run) 1. Capture RTP metrics during a failing call: ◦ Packet loss, jitter, RTT for the SIP leg and for the LiveKit agent leg. ◦ Note the times of audible dropouts and correlate with packet loss spikes. 1. Increase agent-side buffering: ◦ Set 
queue_size_ms
= 1000–2000 ms and test with the same TTS stream. 1. Colocate the agent worker with the LiveKit server (or run in same cloud region/VPC) and re-test. 2. Test with continuous pre-recorded PCM played by the agent (no live TTS) to determine if the bridge path is stable. 3. If possible, record the three audio checkpoints (Twilio ingress, LiveKit SIP output, Agent input) and compare for packet gaps or silent segments. 4. Check CPU and I/O on livekit-server / livekit-sip and the agent during calls for spikes/pressure. If you’d like, I can help with one of these next steps: • Walk through which logs/metrics to capture and how to interpret the RTP stats. • Show the exact agent SDK code snippets to increase 
queue_size_ms
and safely buffer TTS frames. • A short test plan to colocate a worker and compare before/after results. Which diagnostic step would you like to start with? Sources: Agents telephony integration | LiveKit Documentation | Why WebRTC beats WebSockets for realtime voice AI | LiveKit | server-sdk-ruby | livekit/livekit_composite | DeepWiki | Creating SIP Inbound Trunks and Dispatch Rules with Python SDK | livekit/livekit_composite | DeepWiki, with interactive elements
5 Views
Open in Slack
PreviousNext
Powered by