This message was deleted.
# ask-ai
s
This message was deleted.
t
Thanks for reaching out, Durgesh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m looking at possible causes for TTS audio cutting out. Below are the most likely reasons and quick checks you can run to narrow it down. Acknowledging the issue • You said TTS audio is cutting in between — I’ll cover interruption/VAD behavior, streaming/buffering, and other likely causes. Possible causes and quick checks • Interruptions by user speech: Agents allow interruptions by default. If the system detects voice activity it will stop TTS. Check if cut-outs coincide with any user audio; try setting allow_interruptions=False temporarily to see if playback completes. • VAD false positives: A sensitive VAD (e.g., Silero) can detect background noise or brief sounds as speech. Check VAD logs or lower sensitivity / prewarm the VAD model and test again. • Preemptive generation conflicts: With
preemptive_generation=True
, interim transcriptions may trigger regeneration and cancel playback. Disable preemptive generation to test. • TTS streaming / network latency: Streaming TTS providers (WebSocket-based) can have buffering gaps on high latency or unstable networks. Test on a stable network or switch to a non-streaming provider to compare. • Audio buffer/queue underrun or overflow: Internal audio frame queue issues can drop frames. Review agent audio logs for buffer warnings and test with smaller/larger buffer settings if available. • Packet loss / WebRTC issues: Although WebRTC mitigates many problems, client-side packet loss or congestion can affect playback. Check network stats (RTT, packet loss) during playback. Recommended diagnostic steps (in order) 1. Reproduce and correlate: Note timestamps when cut-outs occur and check agent logs for
playback_finished
, interruption events, VAD triggers, or TTS streaming errors. 2. Disable interruptions: Set allow_interruptions=False and retest. 3. Disable preemptive generation: Turn off
preemptive_generation
and test. 4. Test VAD sensitivity: Prewarm the VAD, lower sensitivity, or temporarily disable VAD to isolate the cause. 5. Test TTS provider/network: Try a different TTS provider or run on a known-good network to rule out streaming/latency issues. 6. Check observability: If you have Agent Observability, use traces and transcripts to pinpoint timing of interrupts and playback events. If you’d like, tell me: • Which TTS provider you’re using • Whether you’re using Silero VAD and
allow_interruptions
or
preemptive_generation
settings • Any relevant log snippets or timestamps when cut-outs happen I can then suggest exact settings to change or specific log lines to look for. Sources: Uninterruptable Agent | LiveKit docs | TTS Translator with Gladia STT | LiveKit docs | examples/voice_agents | livekit/livekit_composite | DeepWiki | Live Conversations with AI using ChatGPT and WebRTC | Resolving "InvalidState - failed to capture frame" errors in AudioSource
Thanks — this is helpful. I’ll focus on your provided config and give targeted checks and quick changes to isolate the cut-out. Acknowledging your setup • You’re using Google TTS (streaming), Silero VAD, Soniox STT, Gemini LLM,
allow_interruptions=False
,
preemptive_generation=False
, and
discard_audio_if_uninterruptible=True
at the session level. Targeted checks (quick, in this order) 1. Verify VAD false positives ◦ Check VAD event logs during playback for any VAD-triggered events. Even with
allow_interruptions=False
, a noisy/too-sensitive VAD can cause upstream behavior. Prewarm the Silero VAD and test with quiet background to rule this out. 1. Confirm TTS streaming stability ◦ Since
use_streaming=True
, check network stability and TTS streaming logs for partial/frame errors or reconnects. Test with
use_streaming=False
(non-streaming TTS) to see if cut-outs stop. 1. Validate discard behavior ◦
discard_audio_if_uninterruptible=True
can drop audio when a generation is cancelled. Re-run with
discard_audio_if_uninterruptible=False
to see if audio continues instead of being discarded. 1. Inspect interplay between session and agent flags ◦ You set
allow_interruptions=False
on the agent but
discard_audio_if_uninterruptible=True
and
preemptive_generation=False
on session. Confirm the effective runtime flags (agent vs session) are applied the way you expect and not overridden. 1. Reproduce with simplified pipeline ◦ Temporarily swap STT to a passive stub (no VAD/STT input) and run only TTS playback to confirm whether the issue is TTS or something reacting to live input. 1. Check agent logs for playback lifecycle ◦ Look for
playback_finished
,
playback_cancelled
, or TTS plugin error lines with timestamps that match cut-outs. Suggested quick experiments • Experiment A: Set
use_streaming=False
in Google TTS and retest. • Experiment B: Keep streaming, but set
discard_audio_if_uninterruptible=False
and ensure VAD is prewarmed. • Experiment C: Temporarily replace Silero VAD with disabled VAD or higher threshold to rule out false positives. If you want, share one log snippet covering the timestamps of a cut-out (VAD events + TTS streaming events). With that I can point to the exact event causing the interruption and suggest the exact flag or config change.
Good question — prewarming the Silero VAD ensures the model is loaded once at process start so it’s ready immediately for sessions. Do this once per process and reuse the instance in each session. Quick steps 1. Load the VAD at process startup and store it on the process userdata:
Copy code
def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()

server.setup_fnc = prewarm
1. In your RTC/session entrypoint, retrieve and pass the prewarmed VAD to the agent/session:
Copy code
vad_instance = ctx.proc.userdata["vad"]

session = AgentSession(
    ...,
    vad=vad_instance,
    allow_interruptions=False,
)
Notes • Prewarming avoids per-session model load latency and reduces false starts from VAD initialization. • After you prewarm, confirm the agent is actually using
vad_instance
(not loading a new one) and check VAD logs for expected ready/heartbeat messages. Sources: Repeater | LiveKit docs | VAD Metrics | LiveKit docs | Uninterruptable Agent | LiveKit docs
r
give me full code python
t
Below is a complete Python example that prewarms the Silero VAD at process startup, reuses it in each AgentSession, and integrates your LLM, STT, and streaming Google TTS settings. Adjust imports, paths, and any environment-specific wiring to match your project.
Copy code
# example_agent.py
# Adapt imports to your project layout / package names
from livekit_agent import server, JobProcess, AgentSession  # pseudo-imports — replace with your actual modules
import silero
import soniox
from google import LLM as GoogleLLM, TTS as GoogleTTS
# from sarvam import TTS as SarvamTTS  # optional alternative TTS

# Prewarm function runs once per process before sessions start
def prewarm(proc: JobProcess):
    # Load Silero VAD once and store on proc.userdata for reuse
    proc.userdata["vad"] = silero.VAD.load()
    # Optional: log that VAD is prewarmed
    print("Silero VAD prewarmed and stored in proc.userdata['vad']")

# Assign the prewarm function to the server setup hook
server.setup_fnc = prewarm

# Example handler/entrypoint for creating an AgentSession per connection
def handle_new_session(ctx):
    # Retrieve the prewarmed VAD instance
    vad_instance = ctx.proc.userdata.get("vad")
    if vad_instance is None:
        # Fallback: load here (avoid in production; prewarming should have run)
        vad_instance = silero.VAD.load()

    # Configure components matching your provided setup
    llm = GoogleLLM(
        model="gemini-2.5-flash",
        temperature=0.7,
        max_output_tokens=200,
    )

    stt = soniox.STT(
        params=soniox.STTOptions(
            language_hints=["en", "hi", "ta", "te", "ml", "kn"],
            context="User is talking on the phone with a customer support agent.",
            enable_language_identification=True,
            language_hints_strict=True,
        )
    )

    tts = GoogleTTS(
        language="en-IN",
        voice_name="en-IN-Chirp3-HD-Aoede",
        volume_gain_db=2.0,
        use_streaming=True,
        pitch=0,
        # location="us-central1",
    )

    # Create the Agent (your agent class/init may differ)
    agent = YourAgentClass(
        instructions="Your agent instructions here",
        vad=vad_instance,
        turn_detection=MultilingualModel(),  # replace with your actual turn detection init
        llm=llm,
        tts=tts,
        stt=stt,
        allow_interruptions=False,
    )

    # Create the AgentSession; session-level flags as you specified
    session = AgentSession(
        agent=agent,  # if your API expects agent instance here; otherwise adapt
        discard_audio_if_uninterruptible=True,
        preemptive_generation=False,
    )

    # Start or run the session — adapt to your framework's session run method
    session.run(ctx)  # replace with actual run/serve call

# Wire the new-session handler to the server's connection hook
server.on_new_session(handle_new_session)

# If running as a script
if __name__ == "__main__":
    # Start the server/process loop — replace with your actual server start
    server.start()
Notes and small checks • Replace placeholder imports (
livekit_agent
,
YourAgentClass
,
MultilingualModel
,
server.start()
, etc.) with the actual classes/functions in your codebase. • Ensure
server.setup_fnc = prewarm
is set before the server/process starts so the VAD is loaded once per process. • Confirm the session creation path uses the
vad_instance
from
ctx.proc.userdata
rather than calling
silero.VAD.load()
again. • After deploying, verify prewarm by checking process logs for "Silero VAD prewarmed..." and confirm sessions use the same instance. If you want, I can adapt this to the exact LiveKit SDK class names you’re using — share the module/class names for your Agent, AgentSession, and server if they differ.
r
how to know if user has answered the call, user has rejected the call and user has not answered the call (user is away so no answer after full ring)
t
Thanks — I’ll show how to detect answered, rejected, and no-answer (missed) calls using SIP call status on participants. Which language do you want the code in? I can provide a concise Python example that: • Listens for participant connected/attributes-changed events • Reads
sip.callStatus
and
sip.callID
• Tracks whether
active
was ever seen for a call to distinguish hangup after answer vs hangup without answer Tell me: do you want a full Python snippet now, and are you using LiveKit Agents with SIP participants? Sources: SIP Lifecycle Management Agent | LiveKit docs | Outbound Calling Script | LiveKit docs
r
write python code
t
Below is a concise Python example that listens for SIP participant events in a LiveKit room, inspects
sip.callStatus
, and classifies each call as answered, rejected, or no-answer (missed after ringing). Adjust imports and integration points to match your project's LiveKit client/server library.
Copy code
import asyncio
from collections import defaultdict
from datetime import datetime, timezone

# Replace these with your actual LiveKit SDK imports
# from livekit import Room, Participant, ParticipantKind

# Simple call tracker to determine outcomes
class CallTracker:
    def __init__(self):
        # call_id -> {seen_active: bool, first_seen: datetime, last_status: str}
        self.calls = defaultdict(lambda: {"seen_active": False, "first_seen": None, "last_status": None})

    def update_status(self, call_id: str, new_status: str):
        now = datetime.now(timezone.utc)
        entry = self.calls[call_id]
        if entry["first_seen"] is None:
            entry["first_seen"] = now
        entry["last_status"] = new_status
        if new_status == "active":
            entry["seen_active"] = True

    def finalize_on_hangup(self, call_id: str):
        entry = self.calls.get(call_id)
        if not entry:
            return None
        seen_active = entry["seen_active"]
        last_status = entry["last_status"]
        if last_status == "hangup" and not seen_active:
            return "no_answer"       # rang and then hung up without ever being active
        if last_status == "hangup" and seen_active:
            return "answered_then_hungup"
        if last_status == "active":
            return "answered"
        return last_status

# Instantiate a tracker
tracker = CallTracker()

# Example event handlers (adapt to your SDK event wiring)
async def on_participant_connected(participant):
    # Only handle SIP participants
    if participant.kind != "sip":  # replace with ParticipantKind.PARTICIPANT_KIND_SIP if available
        return

    call_id = participant.attributes.get("sip.callID") or participant.sid or participant.identity
    call_status = participant.attributes.get("sip.callStatus", "unknown")

    tracker.update_status(call_id, call_status)
    print(f"[connected] call_id={call_id} status={call_status}")

    if call_status == "active":
        print(f"✅ Call answered (participant connected active): {call_id}")

    if call_status == "hangup":
        outcome = tracker.finalize_on_hangup(call_id)
        print(f"📴 Call hangup detected on connect: {call_id} -> outcome={outcome}")

# Listen to attribute changes for live updates
async def on_participant_attributes_changed(participant, changed_attributes: dict):
    # Only handle SIP participants
    if participant.kind != "sip":
        return

    call_id = participant.attributes.get("sip.callID") or participant.sid or participant.identity

    if "sip.callStatus" in changed_attributes:
        new_status = changed_attributes["sip.callStatus"]
        tracker.update_status(call_id, new_status)
        print(f"[attr_change] call_id={call_id} sip.callStatus={new_status}")

        if new_status == "active":
            print(f"✅ User has answered the call: {call_id}")

        if new_status == "hangup":
            outcome = tracker.finalize_on_hangup(call_id)
            if outcome == "no_answer":
                print(f"💤 Missed / no answer (ringed then hung up): {call_id}")
            elif outcome == "answered_then_hungup":
                print(f"📞 Answered then hung up: {call_id}")
            else:
                print(f"📴 Hangup outcome: {call_id} -> {outcome}")

# Optional: participant disconnected or removed handler
async def on_participant_disconnected(participant):
    if participant.kind != "sip":
        return
    call_id = participant.attributes.get("sip.callID") or participant.sid or participant.identity
    # If participant disconnects without explicit hangup attribute, treat as hangup
    if tracker.calls.get(call_id):
        tracker.update_status(call_id, "hangup")
        outcome = tracker.finalize_on_hangup(call_id)
        print(f"[disconnected] call_id={call_id} -> outcome={outcome}")

# Example wiring function to attach handlers to your LiveKit Room object
def wire_handlers(room):
    # Replace these with your SDK's event registration methods
    room.on("participant_connected", lambda p: asyncio.create_task(on_participant_connected(p)))
    room.on("participant_updated", lambda p, changes: asyncio.create_task(on_participant_attributes_changed(p, changes)))
    room.on("participant_disconnected", lambda p: asyncio.create_task(on_participant_disconnected(p)))

# Usage example (pseudo-start)
if __name__ == "__main__":
    # room = Room(...)  # create/connect to your LiveKit room
    # wire_handlers(room)
    # run your event loop / room connect lifecycle
    pass
How the code decides outcomes • Answered:
sip.callStatus
becomes
"active"
. • Rejected: final
sip.callStatus
becomes
"hangup"
but the tracker had previously seen
"active"
→ treated as answered then hung up (rejected during ringing would also show hangup without active). • No-answer (missed): final
sip.callStatus == "hangup"
and
active
was never seen for that call ID. If you want, I can adapt this to the exact LiveKit Python SDK classes/method names you use—share the SDK import names and event APIs and I’ll match them.