https://livekit.io logo
Join Slack
Powered by
how can i track AI interrupting user and vice ver...
# ask-ai
r
how can i track AI interrupting user and vice versa ?
t
Thanks for reaching out, Saurabh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about how to track when the AI interrupts the user and when the user interrupts the AI. Below are practical ways you can detect and monitor both directions of interruption in LiveKit Agents, plus configuration knobs and observability tips you can use right away. How to detect "User interrupts AI" • Use the SpeechHandle interruption flag. After calling 
session.say(...)
wait for the handle and check 
handle.interrupted
. If true, the user interrupted the agent. • Ensure 
allow_interruptions
is enabled (default True). This makes the system listen for user speech while the agent is speaking. • Use 
min_interruption_duration
and 
min_interruption_words
to control what counts as a real interruption (helps avoid counting brief noises). • The platform uses VAD + STT: a voiced segment while the agent is speaking that meets your thresholds counts as an interruption. How to detect "AI interrupts user" • The AI should not cut in unless you explicitly trigger it or the end-of-turn detection fires early. • Compare transcript timing vs agent response timing: if the agent begins speaking before the user’s utterance is fully transcribed, that indicates an AI-initiated interruption. • Use the Trace view in Agent Observability to see when end-of-turn detection fired relative to the user audio and STT results — this reveals premature end-of-turn events. Observability & dashboards (recommended) • Use the Transcript tab to scrub audio and pinpoint the exact moment of overlap. • Use the Trace tab to inspect timings for VAD, STT, end-of-turn, and LLM inference — this helps identify whether interruptions were caused by VAD sensitivity, transcription delay, or logic. • Use the Logs tab to find warnings about false positives or latency. Programmatic controls & logging 1. Prevent user interruptions when needed: ◦ Call 
session.say(..., allow_interruptions=False)
to keep the agent from being interrupted. 1. Explicitly trigger or log interruptions: ◦ If you programmatically interrupt (e.g., 
session.interrupt()
), log that event with context (room, session id, timestamps). 1. False-interruption handling: ◦ 
false_interruption_timeout
(default ~2s) determines whether a brief VAD event without words is treated as a false interruption. ◦ 
resume_false_interruption=True
allows the agent to resume if an interruption turns out to be false. Suggested instrumentation for reliable tracking • Emit structured events for: ◦ AgentStartedSpeaking (timestamp, speechHandle id) ◦ AgentStoppedSpeaking (timestamp, reason: finished|interrupted|false_interruption) ◦ UserSpeechDetected (timestamp, STT transcript, confidence) ◦ InterruptEvent (timestamp, initiator: user|agent, context ids) • Correlate these events by speechHandle id, session id, and timestamps to produce reliable interruption metrics (counts, durations, false positives). Quick troubleshooting checklist if you see unexpected interruptions • Check VAD sensitivity and increase 
min_interruption_duration
or 
min_interruption_words
. • Review Trace to see if end-of-turn fired too early (adjust turn-detection parameters). • Review STT latency; delayed transcripts can make it look like the agent interrupted the user. If you’d like, I can: • Provide the minimal code snippet you’d use to check 
handle.interrupted
in your language (ask which SDK you’re using). • Recommend specific parameter values to start with (e.g., 
min_interruption_duration = 0.5s
) based on your use case (short back-and-forth vs. long monologues). • Help design the structured event schema for logging interruptions. Which would be most helpful next: a code example for your SDK, recommended parameter defaults for your use case, or the event schema for logging? Sources: VAD and turn detection configuration guide | LiveKit | Turns overview | LiveKit Documentation | How to detect when an agent has finished speaking | LiveKit | Streamline troubleshooting with Agent Observability | Interrupt User | LiveKit Documentation
r
Provide the minimal code snippet you’d use to check 
handle.interrupted
in python
t
Thanks — I’ll show the minimal Python snippet to call 
session.say(...)
, wait for playout, and check 
handle.interrupted
.
Copy code
# Minimal example: check if the agent's speech was interrupted by the user

# assume `session` is an active LiveKit Agent session object

# Start speaking and receive a SpeechHandle
handle = session.say("Hello, how can I help you today?")

# Wait for speech playout to finish or to be interrupted
await handle.wait_for_playout()

# Check whether the speech was interrupted by the user
if handle.interrupted:
    print("Agent speech was interrupted by the user.")
else:
    print("Agent finished speaking without interruption.")
If you want, tell me which LiveKit Python SDK or async environment you’re using and I can adapt this snippet (for example, adding error handling or logging with timestamps).
r
any session event I can listed to ?
t
Thanks — I’m responding to your question about which session events you can listen to for interruption detection in LiveKit Agents. Key session events to listen for • agent_state_changed — Use this to detect when the agent transitions out of 
speaking
(for example to 
thinking
or 
idle
). A transition from 
speaking
→ another state usually indicates the agent was interrupted or its speech was cancelled. • speech_handle.done / wait_for_playout — Not an event name in all SDKs, but the SpeechHandle completion (via a callback or awaiting 
wait_for_playout()
) combined with the handle’s 
interrupted
flag confirms whether that specific utterance was interrupted. • transcript / stt events — Events delivering partial/final transcripts tell you when user speech occurs; if transcript timing overlaps while the agent is 
speaking
, that implies a user interruption. • custom InterruptEvent (user-defined) — Emit your own structured event when you call programmatic interrupts (e.g., when you invoke 
session.interrupt()
), so you have a clear record of agent-initiated interruptions. Minimal Python example showing 
agent_state_changed
+ SpeechHandle check
Copy code
# assume `session` is an active AgentSession instance

# 1) register a handler for agent_state_changed
@session.on("agent_state_changed")
def on_agent_state_changed(ev):
    # ev.old_state and ev.new_state are available
    if ev.old_state == "speaking" and ev.new_state != "speaking":
        print("Agent left speaking state — possible interruption or completion:",
              "old=", ev.old_state, "new=", ev.new_state)

# 2) start speaking and use SpeechHandle to confirm interruption
handle = session.say("Hello, how can I help you today?")

# await completion (async) or attach a done callback depending on your environment
await handle.wait_for_playout()

if handle.interrupted:
    print("Confirmed: agent speech was interrupted by the user.")
else:
    print("Agent finished speaking without interruption.")
How to combine events for reliable detection • Correlate timestamps and session IDs between: ◦ 
agent_state_changed
transitions, ◦ speech handle completion and 
interrupted
flag, ◦ incoming STT/transcript events. • Emit a structured 
InterruptEvent
in your app when you detect: ◦ User-initiated: transcript arrives while agent was speaking + handle.interrupted true. ◦ Agent-initiated: your code called 
session.interrupt()
or agent started speaking while STT shows user still speaking. Quick tips • Use 
agent_state_changed
to quickly detect potential interruptions in real time. • Use 
SpeechHandle.interrupted
to confirm per-utterance whether the agent was actually cut off. • Use transcript events (partial/final) to determine who started speaking and when. • Log these events together (timestamps, session id, handle id) to analyze false positives and tune parameters. If you want, I can: • Provide a variant of the snippet that uses a done-callback instead of 
await
, • Show how to listen to STT/transcript events in Python and correlate them with the agent state, • Suggest what fields to include in a structured 
InterruptEvent
for logging. Which follow-up would help most? Sources: Turns overview | LiveKit Documentation | How to detect when an agent has finished speaking | LiveKit
r
how can i track Average Pitch (in Hz) ?
t
Thanks — I’m responding to your question about tracking average pitch (in Hz) for speech in your LiveKit interactions. Short answer: LiveKit does not provide a built‑in metric or API to compute pitch (Hz). However, you can capture the audio stream (raw audio) and compute average pitch yourself using a third‑party audio analysis library or service. I can help outline how to do that and what to capture. A workable approach (high level) • Capture the raw audio you want to analyze: ◦ If you have access to the participant audio tracks (server-side or client-side), record or forward the PCM audio frames for the time window you want to analyze. ◦ Include timestamps and identifiers (session/participant id) so results can be correlated with other events. • Run pitch detection on the captured audio: ◦ Use a pitch-detection algorithm or library (e.g., autocorrelation, YIN, or libraries like librosa, pyin, aubio) to estimate fundamental frequency (F0) per frame or per short window. • Compute the average pitch: ◦ Aggregate F0 estimates over your chosen interval (mean, median, or other statistics). ◦ Optionally filter unvoiced frames (no pitch) out before averaging, or treat them according to your use case. • Store and visualize: ◦ Save per-frame F0 and the aggregated metric (average Hz) with timestamps so you can chart trends or correlate with interruptions/turns. Minimal implementation notes (Python suggestions) • Libraries commonly used: ◦ aubio (lightweight, real-time capable) — provides 
pitch
detection functions. ◦ librosa (offline, feature-rich) — provides 
pyin
and other pitch estimators. • Processing flow: a. Acquire audio as PCM (e.g., 16-bit mono, 16kHz or 48kHz). b. Break into short frames (e.g., 20–40 ms) with overlap. c. For each frame, run the pitch detector to get F0 (Hz) or a voiced/unvoiced decision. d. Remove or mark unvoiced frames (F0 = 0 or None). e. Compute average over frames: mean(F0) or median(F0). Example libraries and links • aubio: real-time pitch detection (good for streaming scenarios). • librosa: 
librosa.pyin
for offline, high‑quality pitch extraction. (If you want, I can include exact pip install commands and a short code snippet for one of these libraries.) Questions to narrow the recommendation • Do you have access to raw audio frames server-side, or do you prefer client-side processing in the browser? • Do you need real‑time (streaming) pitch estimates or offline/near‑real‑time analysis? • Which languages/environments do you prefer (Python, JavaScript/browser, Node.js)? • Do you want to exclude unvoiced frames from the average or include them as zeros? Next step Tell me which environment and latency requirement you have (server vs browser, real‑time vs offline, Python vs JS). I’ll provide a minimal, copy‑paste code snippet using aubio or librosa tailored to your setup and show how to handle voiced/unvoiced frames and compute the average pitch. Sources: Live Conversations with AI using ChatGPT and WebRTC | Text-to-speech (TTS) models overview | LiveKit Documentation | Turns overview | LiveKit Documentation
r
agent to agent conversation for testing scenrios
t
Thanks — I’m responding to your request about agent-to-agent conversations for testing scenarios. Short answer: you can have one agent (an evaluation agent) interact with another agent (the agent under test) by changing the session participant filter so the evaluation agent subscribes to agent-kind participants. Below is a concise guide and minimal Python examples to get you started. What to change (concept) • By default agents ignore audio from other agents. Override that behavior using 
RoomInputOptions
(participant filter) or 
RoomOptions
(participant kinds) so the evaluation agent will receive audio from 
PARTICIPANT_KIND_AGENT
. • Run both agents in the same LiveKit room. The evaluation agent can be scripted to play prompts, listen, and then grade or log the agent-under-test responses. Minimal Python snippets 1) Start an evaluation agent session that accepts agent participants
Copy code
from livekit.agents import RoomInputOptions

# assume session is an AgentSession or similar API object
await session.start(
    agent=evaluation_agent,
    room=room,  # your room reference
    input_options=RoomInputOptions(
        participant_filter=lambda p: True  # accept all participants including agents
    ),
)
Alternate: explicitly allow agent participant kind
Copy code
from livekit import rtc
from livekit.agents import RoomOptions

await session.start(
    agent=evaluation_agent,
    room=room,
    room_options=RoomOptions(
        participant_kinds=[rtc.ParticipantKind.PARTICIPANT_KIND_AGENT]
    ),
)
2) Example evaluation agent behavior (high level) • Script the evaluation agent instructions to: ◦ Send a test prompt (e.g., “I need help with my order”). ◦ Wait for the agent under test to respond (use 
SpeechHandle
or transcript events). ◦ Grade or log the response using a local tool function (e.g., a 
grade_response
tool). • Use STT/TT S or LLM tools inside the evaluation agent as needed for analysis. Simple evaluation agent skeleton (pseudo-Python)
Copy code
class EvaluationAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="""
                You are an evaluation agent. Ask test questions, wait for responses,
                then call the grade_response tool to score the reply.
            """,
            tools={
                "grade_response": llm.tool({
                    "description": "Grade the agent's reply",
                    "execute": async lambda params: <http://logger.info|logger.info>(f"graded: {params}")
                })
            },
            stt=..., tts=..., llm=..., vad=...
        )

    async def run_test(self):
        handle = self.session.say("I need help with my order")
        await handle.wait_for_playout()
        # wait for agent under test reply (listen for transcript or SpeechHandle)
        # then call grade_response with evaluation metrics
Testing flow 1. Deploy the agent under test in the room. 2. Start the evaluation agent with the participant filter allowing agents. 3. Evaluation agent issues prompts and listens. 4. Evaluation agent grades/logs the responses and records metrics (timestamps, transcripts, audio). 5. Aggregate results for analysis. Best practices • Correlate by session/participant IDs and timestamps so you can trace each exchange. • Log SpeechHandle ids, transcript events, and any grading output. • Use deterministic prompts and grading criteria for repeatable tests. • If you need human-readable reports, emit structured events (prompt_id, response_id, clarity/helpfulness scores, notes). • Consider isolating test runs in dedicated rooms so evaluation traffic doesn’t affect production agents. If you want, I can: • Provide a complete, runnable example from the LiveKit examples repo tailored to your SDK version. • Show a concrete grading tool implementation (with schema) the evaluation agent can call. • Provide a variant that runs both agents locally for automated CI tests. Which follow-up would you like: a runnable example, a grading tool implementation, or instructions for automating tests in CI? Sources: examples/voice_agents | livekit/livekit_composite | DeepWiki | How to Test Your Agent Using Another Agent | LiveKit
r
livekit ivr detection
t
Thanks — I’m responding to your question about LiveKit IVR detection. Short answer: LiveKit supports IVR-related features you can use to detect and interact with automated phone systems — key items are the 
ivr_detection
option and DTMF send/receive support. Below is a concise summary of what’s available and next steps you can take. What LiveKit provides for IVR • ivr_detection: A session option you can enable to help detect when the agent has reached an automated IVR system (default is False). This helps agents distinguish IVR prompts from human speech. • DTMF receiving: LiveKit emits 
dtmf_received
events when callers press keys. Each event contains the digit (0–9, *, #, A–D), the participant, and a timestamp. • DTMF sending: Agents can send tones to navigate IVR menus using the prebuilt 
send_dtmf_events
tool or equivalent APIs (supports RFC 2833, SIP INFO, and in-band where applicable). • IVR navigation tools: Tasks like 
GetDtmfTask
are available to collect keyed input or spoken digits during IVR flows. • Call flow features: Outbound/inbound SIP integrations, call transfer options, and tools for ending or transferring calls are supported and useful for IVR scenarios. How to use these features (practical steps) 1. Enable IVR detection for the session where the agent might hit an IVR: ◦ Set 
ivr_detection=True
on the AgentSession when starting calls. 1. Listen for DTMF events: ◦ Subscribe to 
dtmf_received
events to capture keypad input from the remote IVR or caller. 1. Send DTMF to navigate menus: ◦ Use 
send_dtmf_events
or 
send_dtmf_code(digit)
from your agent logic at the right time to press menu options. 1. Use STT and prompts: ◦ Combine speech detection (STT) and DTMF to decide when to send tones or wait for a human. 1. Log and correlate: ◦ Record timestamps, participant IDs, transcripts, and DTMF events so you can replay or analyze IVR interactions. Example use cases • Automated IVR navigator: Agent places outbound call, listens for prompts, sends DTMF to make selections, and detects when a human answers. • IVR detection before conversation: Enable 
ivr_detection
to avoid wasting resources on automated systems or to trigger alternative handling. • Collecting keypad input: Use 
GetDtmfTask
to reliably collect digits from users during a call. Questions to recommend a concrete setup • Are you building inbound or outbound call flows? • Do you need to navigate IVR menus automatically (agent presses keys), or only detect IVR vs. human? • Are you using SIP trunks or WebRTC clients for these calls? • Do you need sample code (Python/JS) showing how to enable 
ivr_detection
, send DTMF, and listen for 
dtmf_received
? Tell me which of the above matches your goal and which SDK/language you prefer (Python or JavaScript). I’ll provide a minimal code example and recommended flow for your scenario. Sources: Agent session | LiveKit Documentation | examples/dtmf | livekit/livekit_composite | DeepWiki | Prebuilt tools | LiveKit Documentation | send_dtmf_events | LiveKit Documentation | Building an Automated IVR Menu Caller | LiveKit Documentation
r
how to handle ssml break for manual pause
t
Thanks — I’m responding to your question about using SSML 
<break>
tags to add manual pauses in TTS. Short answer: Use SSML’s 
<break>
tag inside a 
<speak>
element. You can specify an explicit duration with 
time
(e.g., "500ms" or "1s") or a semantic pause with 
strength
("weak", "medium", "strong", etc.). Also ensure SSML mode is enabled for your TTS provider in LiveKit. Minimal SSML examples • 500 ms pause:
Copy code
<speak>
  Hello there. <break time="500ms"/> Let me continue.
</speak>
• 1 second pause:
Copy code
<speak>
  Please wait. <break time="1s"/> Thank you for waiting.
</speak>
• Semantic strength (provider-dependent):
Copy code
<speak>
  Step one. <break strength="strong"/> Step two.
</speak>
Provider configuration notes • Some TTS backends require SSML mode to be enabled: ◦ ElevenLabs: set 
enable_ssml_parsing=True
when creating the TTS plugin. ◦ Amazon Polly: use 
text_type="ssml"
. • Behavior and supported tags/attributes can vary by provider. Always test the exact pause durations and strength values with your chosen TTS engine. Best practices • Wrap content in a top-level 
<speak>
element. • Use 
time
for precise pauses (recommended for consistent timing). • Use 
strength
for more semantic control when exact timing is not required. • Keep pauses reasonable for natural-sounding speech (start with 300–1000ms for short pauses). • Test across the TTS providers you plan to support — some may slightly alter actual pause length. If you tell me which TTS provider (e.g., ElevenLabs, Amazon Polly, Google Cloud, Azure) and which SDK/language you’re using, I’ll give a concrete example showing how to enable SSML and pass the SSML string to the LiveKit TTS plugin. Sources: ElevenLabs TTS plugin guide | LiveKit Documentation | Amazon Polly TTS plugin guide | LiveKit Documentation | Resemble AI TTS plugin guide | LiveKit Documentation | Azure Speech TTS plugin guide | LiveKit Documentation | Speechify TTS plugin guide | LiveKit Documentation
2 Views
Open in Slack
PreviousNext
Powered by