Can we determine when livekit thinks it is talking to an LVR vs talking to a person? cc @acoustic-lunch-30313

Thanks for reaching out, James. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Server side. We are running into issues where the agent is talking to the IVR, then being transfered to a human and missing the leading message about why they are calling. We could try something about repeating yourself whenever transfered but unfortunately people transfer between each other where we don't want to repeat our long intro. We were looking to see if livekit had a way to determine if it was in an IVR flow/ talking to a robot.

Thanks — this is helpful context. I’m responding to your point about server‑side Agents getting routed through IVRs and then transferred to humans, which causes the agent to miss the caller’s leading message. Short answer: LiveKit does not have a built‑in IVR/LVR classifier. The available server‑side signals (VAD, end‑of‑utterance, silence detection) can be used as inputs, but you’ll need to combine them with custom logic or an external classifier to reliably detect IVR/robot flows. Practical server‑side approaches you can try (mix-and-match as needed) • Silence + timing heuristics ◦ If the call exhibits a long silence before any speech after your prompt, or long repeating silences/regular timing patterns, flag it as likely automated. ◦ If a response starts immediately at a consistent offset after prompts (and is very short), that’s another IVR clue. • Pattern detection for prerecorded audio ◦ Detect the presence of a beep tone or a fixed greeting pattern by matching short audio fingerprints (e.g., using simple spectral features or hashing). ◦ If you have examples of the IVR prompts, create lightweight templates to match against incoming audio snippets. • Turn-length and variability heuristics ◦ Automated systems often have very uniform turn lengths and phrasing. If initial turns are very short and repeat across calls, treat them as automated. • Multi-signal scoring ◦ Combine VAD events, silence durations, immediate/short responses, and tone detection into a single confidence score. Apply thresholds to decide “likely IVR” vs “likely human.” • External classifier service ◦ Send a short buffer (first few seconds) of audio to a third‑party model trained to classify human vs machine speech, and use the result server-side to decide behavior. • Transfer-aware state handling (to avoid unwanted long intros) ◦ When a transfer occurs, check your IVR flag: ▪︎ If flagged as IVR earlier in the call, treat the next connected party as “probable human” only after you detect a human signal (e.g., a sufficiently long, natural reply or lack of consistent IVR timing). ▪︎ Alternatively, play a short recap (1–2 sentence summary) only when your IVR confidence is high — avoid repeating long intros on every transfer. Low-effort implementation patterns 1. Conservative recap on suspected IVR transfers: ◦ If IVR flag is true, play a brief summary: “This call is about [topic].” Wait for a human confirmation. If not confirmed, fall back to full intro after N seconds. 1. Two-stage playback: ◦ Stage 1: short one-sentence recap immediately after transfer. ◦ Stage 2: if no human confirmation or further context is detected within M seconds, optionally play the full intro. 1. Post-transfer human-detection latch: ◦ After transfer, wait until you observe X seconds of natural speech from the callee (longer, variable turns) before assuming you’re talking to a human; until then keep reprompts short. Questions so I can give concrete thresholds or sample pseudocode 1. How are you detecting transfers server‑side? (Is there a call state event you receive when a transfer happens?) 2. How long is your current full intro, and how short would a “brief recap” be acceptable (e.g., 5–10 seconds)? 3. Do you have access to sample IVR audio (or a list of known carriers/IVR vendors) that we could use to build simple fingerprint rules? 4. Are you comfortable sending audio to a third‑party classifier, or must everything stay in your environment? If you tell me which pattern you prefer (conservative recap, two‑stage playback, or external classifier), I’ll provide concrete server‑side logic and example thresholds (with pseudocode) you can plug into your LiveKit Agent flow.

In that case what does the ivr_detection=True do?

Thanks — I’m following up on your question about what

ivr_detection=True

does. Short answer:

ivr_detection

is a boolean AgentSession option that, when set to

True

, enables LiveKit’s IVR/answering‑machine detection capabilities for server‑side Agents. It does not guarantee perfect classification, but it enables detection signals and behaviors that help your agent handle IVR flows more intelligently. What enabling

ivr_detection=True

provides • Activates IVR/answering‑machine detection logic at the session level so the agent can look for cues of automated systems (for example, structured prompts, lack of natural speech, repetitive audio patterns). • Lets your agent take IVR‑specific actions such as navigating IVR menus (DTMF), detecting voicemail greetings, and handling call termination or message leaving flows more appropriately. • Is intended for outbound scenarios where the agent may encounter phone systems before reaching a human. What it is not • It is not a perfect human-vs‑robot classifier that always identifies LVR/IVR with 100% accuracy. You should treat its output as a signal that can be combined with other logic. • It does not automatically perform every IVR interaction without additional logic; enabling detection lets you integrate IVR responses (like DTMF or alternative prompts) or routing decisions in your session flow. How teams typically use it (recommended patterns) • Combine

ivr_detection=True

with silence/VAD signals and timing heuristics to form a confidence score before deciding to repeat a long intro. • If the session is flagged as IVR, trigger a short recap or IVR navigation routine rather than the full intro. • After a transfer, re-check for human signals (longer, variable turns) before replaying long content — use the IVR flag to decide whether to play a short recap first. If you’d like, I can: • Provide example server‑side pseudocode showing how to read the IVR flag and combine it with VAD/silence thresholds, or • Suggest concrete thresholds and a two‑stage playback flow (short recap → full intro) you can implement with your AgentSession. Which would you prefer? Sources: Agent session | LiveKit Documentation | Building an Automated IVR Menu Caller | LiveKit Documentation

Can you provide

Suggest concrete thresholds and a two‑stage playback flow (short recap → full intro) you can implement with your AgentSession.

Thanks — I’ll provide concrete thresholds plus a clear, server‑side two‑stage playback flow you can drop into your AgentSession. I’m responding to your request for “concrete thresholds and a two‑stage playback flow (short recap → full intro)” to handle transfers from IVR to humans. Summary of the approach • Stage 1: Play a short recap (<= 5s) immediately after transfer. • Listen briefly (1.5s) for the callee to interrupt or speak naturally. • If interrupted or the system detects natural human speech, skip the long intro. • If no human signal, play the full intro (15–20s) but still allow interruption. • Use the AgentSession IVR flag and VAD/silence signals as part of the decision logic. Suggested thresholds • Short recap max duration: 3–5 seconds. • Full intro max duration: 15–20 seconds. • Interruption window after recap: 1.5 seconds (treat any user speech of >= 1.5s as a human signal). • User‑away / silence timeout to detect dead line/voicemail: 5.0 seconds. • Minimum speech length to consider “human” after transfer: 1.5–2.0 seconds of continuous speech. • Confidence scoring: require 2 supporting signals (e.g., VAD activity + variable turn length) before classifying as human. Example flow (high level) 1. On transfer event: ◦ Wait 0.5–1.0s to discard trailing IVR audio. ◦ Check

ivr_detection

flag. If true, treat system as “likely IVR” initially. 1. Stage 1 — Short recap: ◦ Play a short, one‑line recap (<=5s) with interruptions allowed. ◦ Start a 1.5s listening window for user speech. ◦ If user speaks >=1.5s, assume human and stop; continue normal dialog. 1. If no human speech detected: ◦ Play full intro (15–20s) with interruptions allowed. ◦ After full intro, wait for human speech (use same 1.5–2.0s minimum). 1. Post‑playback: ◦ If still no human signal after full intro and repeated attempts, mark as voicemail/IVR and take fallback action (leave message, hang up, or schedule retry). Server‑side pseudocode (conceptual) • This pseudocode assumes you can: ◦ read an

ivr_detection

boolean, ◦ call

session.say(text, allow_interruptions=True)

returning a

handle

with

wait_for_playout()

and interruption metadata, ◦ listen for VAD / user_start events and measure continuous speech duration, ◦ use timeouts.

# Configurable thresholds
SHORT_RECAP_MAX = 5.0
FULL_INTRO_MAX = 20.0
INTERRUPTION_WINDOW = 1.5
SILENCE_TIMEOUT = 5.0
MIN_HUMAN_SPEECH = 1.5

async def on_transfer(session):
    # Small buffer to ignore trailing IVR audio
    await asyncio.sleep(0.8)

    # Use ivr_detection as an initial signal
    ivr_flag = session.agent_session_options.get("ivr_detection", False)

    # Stage 1: short recap
    recap_text = "You are now connected to a specialist. One moment please."
    recap_handle = session.say(text=recap_text, allow_interruptions=True, max_duration=SHORT_RECAP_MAX)
    # Wait for playout to finish or interruption
    await recap_handle.wait_for_playout()

    # If user interrupted during recap, treat as human (check interrupted speech length)
    if recap_handle.interrupted:
        if session.last_user_speech_duration >= MIN_HUMAN_SPEECH:
            return  # user engaged; continue conversation

    # Short listening window for immediate user response
    if await wait_for_user_speech(timeout=INTERRUPTION_WINDOW, min_duration=MIN_HUMAN_SPEECH):
        return  # human confirmed

    # Stage 2: full intro (play only if needed)
    full_intro_text = "Hi, I'm Alex from Billing. I see there was an issue with your last payment. How can I help today?"
    intro_handle = session.say(text=full_intro_text, allow_interruptions=True, max_duration=FULL_INTRO_MAX)
    await intro_handle.wait_for_playout()

    # After full intro, wait for human speech for a short period
    if await wait_for_user_speech(timeout=5.0, min_duration=MIN_HUMAN_SPEECH):
        return  # human confirmed

    # If still no human speech, treat as IVR/voicemail fallback
    handle_voicemail_or_retry(session)

Helpful implementation notes • Allow interruptions on both recap and intro so a real person can jump in quickly. • Use

min_interruption_duration

or equivalent VAD metric to avoid reacting to brief noise or DTMF tones. Require ~1.5s of continuous speech to count as human. • Keep a session flag like

played_recap

to avoid repeating the recap on every internal transfer between humans. • If

ivr_detection

was true earlier in the call, bias toward short recap first; if it was false, consider skipping recap unless transfer count > 1. • Consider a short confirmation prompt after recap if you want to minimize risk of missing the leading message, e.g., “Can you confirm you can hear me?” (very short, low friction). • Pre-generate or cache TTS audio for recap/intro to reduce latency during transfer. Questions to fine‑tune thresholds for your environment 1. Do you receive an explicit transfer/call_state event from your telephony integration when transfer happens? If so, we can tie this flow directly to that event. 2. How long is your current full intro today (seconds)? That helps pick the exact full_intro cap. 3. Do you see lots of short exchanges between humans that would be disrupted by a recap? If yes, raise the barrier to play the recap (require ivr_flag true + transfer count > 1). If you want, I can: • Provide a concrete Node.js or Python code sample tailored to the AgentSession API you’re using, or • Recommend exact values for the speech detection thresholds after you tell me your current full intro length and whether you can access transfer events. Which would you prefer? Sources: Agent speech and audio | LiveKit Documentation | Uninterruptable Agent | LiveKit Documentation | VAD and turn detection configuration guide | LiveKit | Silero VAD plugin | LiveKit Documentation | Handling payments with PCI compliance | LiveKit

How will this help us solve the issue if we face an automated voice where we need to send a DMTF code to handle a phone tree?

Great — you’re asking how to handle an automated IVR that requires DTMF navigation so the agent can reach the human and not miss the caller’s leading message. I’ll explain how to combine IVR detection, DTMF sending/collection, and the two‑stage playback flow so the agent can navigate phone trees and avoid losing important context. How this solves your problem (high level) • Enable

ivr_detection=True

so the session treats the call as “likely IVR” when automated cues are present. This biases your agent to use navigation logic instead of immediately playing a long intro. • While inside the IVR, send DTMF tones programmatically (to select menu options) and/or collect DTMF input from the remote system (if you need digits). • After you finish IVR navigation and the callee is transferred, use the two‑stage playback (short recap → full intro) plus VAD/speech checks to avoid missing the caller’s leading message or over‑repeating the long intro. Concrete components and recommended behavior 1) Enable IVR detection on the session • Set

ivr_detection=True

in the AgentSession options to enable IVR/answering‑machine signals. Treat that flag as an initial hint, not a final decision. 2) Navigate IVR menus by sending DTMF • Use the session/room participant method to publish DTMF tones when you need to press menu keys (e.g., press "1" for sales). • Send tones programmatically after detecting the IVR prompt or by instruction from your workflow/LLM. 3) Collect DTMF input from the remote side • Use a DTMF collection helper (e.g.,

GetDtmfTask

) to listen for keypad tones and validate input when you need digits from the caller or to confirm menu choices. 4) Two‑stage playback after transfer (so you don’t miss caller context) • Stage 1 (short recap): Play a 3–5s recap immediately after transfer. Allow interruptions. • Listen for 1.5–2.0s of continuous human speech; if detected, skip the full intro. • Stage 2 (full intro): If no human detected, play the full intro (cap 15–20s). Allow interruptions here too. • If still no human speech after the full intro, mark as voicemail/IVR fallback (leave message, hang up, retry). Suggested thresholds (practical) • Short recap: <= 5s • Full intro: 15–20s • Interruption window after recap: 1.5s • Minimum continuous speech to count as human: 1.5–2.0s • Silence timeout for voicemail detection: 5.0s • Require 2 supporting signals (VAD + variable turn length or DTMF absence) before flipping classification to “human” Example flow tying it together (concept) 1. Outbound call answers → IVR detection may be true. 2. If

ivr_detection

or audio patterns indicate IVR: ◦ Run DTMF navigation steps (send tones as needed). ◦ Optionally collect DTMF from the remote system if you expect keypad responses. 1. When you detect the IVR has finished (silence patterns, known prompt end, or transfer event): ◦ Wait ~0.8–1.0s to drop trailing IVR audio. ◦ Run two‑stage playback: recap → listen 1.5s → full intro if needed. 1. If the call is transferred and you detect human speech during the recap, skip full intro and continue conversation. If not, play full intro once and then fallback. Minimal code sketch (conceptual) • Send DTMF: ◦ Call

session.room.local_participant.publish_dtmf(digit='1')

when you need to press 1 for a menu option. • Collect DTMF: ◦ Use

GetDtmfTask

to wait for numeric input and confirm. • Two‑stage playback: ◦ Play recap with

session.say(..., allow_interruptions=True)

, wait up to 1.5s for

user_started_speaking

event and measure continuous speech. If not present, play full intro. Practical tips and edge cases • If IVR plays a long menu before transfer, use audio fingerprinting (if possible) to detect the menu end and only then do recap logic. • Some IVRs use short tones or very brief speech; require ~1.5s continuous speech to reduce false positives from tones/DTMF. • Cache or pre‑synthesize recap/intro TTS to avoid latency during transfer. • Maintain a session flag (e.g.,

played_recap

) to avoid repeating recap between internal human transfers. • If your environment must avoid sending audio to external services, keep detection heuristic‑based (VAD + timing) rather than ML classification. If you’d like, I can: • Provide a concrete Python or Node.js snippet using your AgentSession API showing how to call

publish_dtmf()

, run

GetDtmfTask

, and wire up the two‑stage playback with VAD events; or • Draft a decision tree that maps common IVR flows (press 1 → transfer, press 2 → human) to automated DTMF actions you can plug into your workflow. Which would be most helpful — a code example for your language (Python/Node), or a decision tree mapping for your typical IVR targets?