is there any relation bw ivr_detection as true and llm_node, for every utterance by user do i get a flag if the utterance was ivr system or not

Thanks for reaching out, Harsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

how does it work realtime then, my usecase involves talking to ivr first then to human rep in the same call also what exactly ivr_dectection helps with then can you provide detailed example

Thanks — I’ll explain how

ivr_detection

works in real time for your use case (IVR first, then a human rep), what it actually helps with, and give a concrete example and recommended pattern. Short summary of behavior •

ivr_detection=True

runs session-level detection to decide when the agent is talking to an automated IVR versus hearing natural human speech. • It does NOT add a per-utterance “is_IVR” flag into messages sent to the

llm_node

. The LLM sees the chat context, but IVR detection is handled by the agent runtime logic and tool gating rather than by tagging every user utterance for the LLM. • The practical effect: tools that would be harmful on a human (e.g.,

send_dtmf_events

) are allowed only while the runtime believes the far end is an IVR; when the runtime detects human-like speech it disables those tools and switches to natural conversation mode. How it works in real time (conceptual) • The runtime analyzes audio patterns (prompts, timing, repetitive/robotic phrasing, long silence patterns typical of IVRs). • When it identifies IVR characteristics it marks the session state internally as “IVR” and enables IVR-specific actions (DTMF sending, automated menu navigation). • When the speech characteristics change to natural/human patterns, the runtime flips state to “human” and disables DTMF output so you don’t send tones to a person. • This detection is continuous, so the call can transition IVR → human during the same session and the runtime changes behavior automatically. Detailed example (call flow) 1. Outbound call placed to company. IVR answers: “Welcome. For billing press 1…” 2. Runtime detects robotic prompt patterns and sets internal IVR state (because

ivr_detection=True

). 3. Agent issues DTMF to navigate: call uses

send_dtmf_events

to press menu choices. DTMF is allowed because IVR state is true. 4. IVR routes the call to a live agent. Speech becomes natural (interruptions, shorter pauses). Runtime detects human-like audio and flips state to human. 5. DTMF is now disabled automatically; agent switches to TTS/voice natural conversation and continues the call with the human rep. What

ivr_detection

helps with (practical benefits) • Prevents sending audible DTMF tones to humans (avoids broken UX). • Safely enables automated navigation of phone trees when interacting with IVRs. • Automates mode switching so you don’t have to hard-code when to enable/disable DTMF or similar IVR tools. • Works in real time so a single call can move from IVR automation to live conversation without manual state flips. Limitations / important notes • There is no documented per-utterance IVR flag delivered to the

llm_node

. If you need the LLM to be explicitly aware of “this utterance came from an IVR,” you will need to inject that context yourself into the prompt or system messages. • IVR detection focuses on identifying automated systems; it does not “confirm” human presence beyond switching to human mode when speech looks human-like. • Use DTMF and IVR tools (e.g.,

send_dtmf_events

,

GetDtmfTask

) only in flows guarded by

ivr_detection=True

. If you want, I can: • Show a short code pattern that injects an IVR indicator into the LLM context when the runtime detects IVR (so the model explicitly knows the source), or • Provide a minimal example sequence using

send_dtmf_events

and how to handle the runtime state transitions in your agent logic. Which would you prefer? Sources: Agent session | LiveKit Documentation | send_dtmf_events | LiveKit Documentation | examples/dtmf | livekit/livekit_composite | DeepWiki | GetDtmfTask | LiveKit Documentation, with interactive elements

if ivr_detection is true and the next utterance is detected as from ivr , then instead of llm_node being used the send_dtmf_events and other ivr tools are being called ?

Short answer: No — ivr_detection doesn't automatically bypass the

llm_node

. It gates IVR tools (like

send_dtmf_events

) so they only run when the runtime believes the far end is an IVR, but the

llm_node

still receives/handles chat context unless your agent logic explicitly chooses to skip or route messages differently. How it typically behaves • When

ivr_detection=True

, the runtime sets an internal IVR/human state based on audio patterns. • IVR-specific tools are allowed only while the runtime is in the IVR state. That prevents DTMF from being sent to humans. • The

llm_node

is not forcibly replaced — you can still have the LLM process transcripts or decide actions. The runtime simply enables/disables tools; whether you call those tools, call the

llm_node

, or do both is determined by your agent's decision logic. Practical pattern (recommended) 1. Let

ivr_detection

control tool availability (DTMF enabled only in IVR state). 2. In your agent logic, on each new user utterance: ◦ Check current IVR state. ◦ If IVR state: run IVR handlers (DTMF sequences, menu parsers). Optionally also log or pass a short note to the LLM (e.g., “Detected IVR — took DTMF action”). ◦ If human state: route utterance to the

llm_node

for natural response generation. If you want, I can provide a minimal code pattern showing that decision flow and an example prompt snippet to inject an “IVR detected” note into the LLM context. Which would help more? , with interactive elements

sure, also explain the edge cases

Thanks — I’ll give a concise decision-flow code pattern you can adapt, show how to inject an IVR indicator into the LLM context, and list key edge cases and recommended handling. Code pattern (pseudo/Python-style) 1. Maintain runtime IVR state (set by platform detection). 2. On each transcript/utterance: ◦ Check IVR state. ◦ If IVR state: run IVR handlers (DTMF, menu parsing). Optionally notify LLM with an injected note. ◦ If human state: send transcript to

llm_node

. Example (minimal)

# state kept by your agent runtime (updated by ivr_detection)
is_ivr = session.ivr_state  # bool updated in real time

async def handle_utterance(transcript, metadata):
    global is_ivr

    # Optionally refresh is_ivr from runtime event
    is_ivr = session.ivr_state

    if is_ivr:
        # Use IVR tools
        await send_dtmf_events(sequence="1")
        # Optionally log or pass a short context note to LLM
        system_note = "NOTE: runtime detected IVR; executed DTMF navigation."
        await llm_node.call(system_message=system_note, user_message=transcript)
    else:
        # Human — route to LLM for natural response
        await llm_node.call(user_message=transcript)

How to inject an explicit IVR indicator into LLM context • Add a short system message or prefix when is_ivr is true: ◦ System: "Runtime detected automated IVR for this utterance. Prefer DTMF/menu actions; do not ask clarifying human questions." ◦ Then send the user transcript. • Remove or change the system note when is_ivr flips to human. Edge cases and recommended handling • False positives (IVR detected but actually human) ◦ Risk: sending DTMF to a human if your logic does so automatically. ◦ Mitigation: never send audible DTMF automatically without a short verification (e.g., confirm IVR prompts or wait for clear IVR-like prompt patterns). Prefer conservative thresholds and instrument logging to tune detection. • False negatives (IVR not detected but it is an IVR) ◦ Risk: agent tries to converse naturally and fails to navigate menus. ◦ Mitigation: allow manual/heuristic triggers for DTMF (e.g., if transcript contains “press 1”, treat as IVR), or provide a fallback “attempt DTMF” action that can be retried when stuck. • Rapid state flips (IVR ↔️ human quickly, e.g., short IVR prompt then human interruption) ◦ Risk: race conditions sending a DTMF tone while human speaks, or missing a needed DTMF because state changed. ◦ Mitigation: debounce state changes (require N consecutive frames/utterances to flip state) or add a short safety window after a flip during which DTMF is suppressed. • Mixed segments in one utterance (part IVR prompt + human overlap) ◦ Risk: ambiguous routing. ◦ Mitigation: split long audio into smaller utterances, prefer the dominant pattern, and surface ambiguous cases to LLM with an “ambiguous source” flag so it can respond cautiously. • Transfer events (IVR routes call to human) ◦ Behavior: runtime should detect change and disable DTMF. ◦ Recommended: when a transfer is detected, clear IVR-specific queued actions and inject a system message to LLM: "Call transferred to human — resume natural conversation." • DTMF audible leakage to humans ◦ Risk: if detection logic incorrectly allows DTMF, humans hear tones. ◦ Mitigation: only allow

send_dtmf_events

when runtime confirms IVR state and optionally after recognizing explicit IVR prompts. Practical tips • Log ivr_state transitions and tool invocations to tune detection thresholds. • Use short system notes to the LLM rather than full per-utterance flag fields if the platform doesn’t expose per-message IVR flags. • Provide a manual override (operator or policy) to force IVR vs human mode for problematic numbers/flows. If you want, I can produce a slightly more complete code example that includes debounce logic for state flips and an example system-message template for the LLM. Which would you prefer? , with interactive elements

sure explain the code using custom agent, llm_node and tts_node for the below usecase if ivr is detected , classification is done to either use dtmf or tts else tts is used for human but a flag that the utterance was from human not ivr is required

Thanks — I’ll give a concise, runnable pattern you can adapt: a custom agent that uses the session’s IVR detection state, classifies IVR responses to DTMF vs TTS, uses TTS for human replies, and ensures a

via_human

flag is passed into the TTS/LLM context. I’ll also call out key edge-case handling. Assumptions •

ivr_detection=True

is enabled on the session so the runtime updates an IVR state you can read (session.info.is_ivr_detected or similar). • You have access to

send_dtmf_events

as a tool. • This is a pattern — adjust APIs/names to match your SDK version. 1) High-level flow • On each incoming utterance (STT transcript): ◦ Read runtime IVR state. ◦ If IVR: classify whether to respond with DTMF or TTS. ▪︎ If DTMF: call

send_dtmf_events

. ▪︎ If TTS: synthesize short TTS suitable for IVR. ◦ If Human: send transcript to

llm_node

for natural response; mark response with

via_human=True

so

tts_node

knows it’s for a human. 2) Minimal example (Python-like pseudocode)

class IVRAwareAgent(Agent):
    def __init__(self):
        super().__init__(instructions="Adapt to IVR or human", tools=[send_dtmf_events])
        self._is_ivr = False
        self._responder = "tts"           # "dtmf" or "tts"
        self._utterance_origin = "unknown"  # "ivr" or "human"

    async def on_new_transcript(self, transcript):
        # Called for every user utterance
        # Refresh IVR state from session/runtime
        self._is_ivr = getattr(self.session.info, "is_ivr_detected", False)

        if self._is_ivr:
            self._utterance_origin = "ivr"
            # classify whether to use dtmf vs tts for this IVR prompt
            self._responder = await self.classify_for_ivr(transcript)
            if self._responder == "dtmf":
                digits = self._extract_digits_or_map_intent(transcript)
                # send DTMF only when runtime allows (ivr_detection gated)
                await send_dtmf_events(events=digits, room=self.session.room)
                # Optionally log and notify LLM with a short system note
                await self._notify_llm_ivr_action(transcript, "dtmf", digits)
                return
            else:
                # IVR but choose TTS: synthesize short TTS response
                self._utterance_origin = "ivr"
                out_text = await self._compose_ivr_tts(transcript)
                await self._synthesize_and_play(out_text)
                return
        else:
            # Human path: pass to LLM and mark via_human=True
            self._utterance_origin = "human"
            llm_response = await self._call_llm_with_origin(transcript, origin="human")
            # LLM returns text; TTS node will see origin and use human voice/style
            await self._synthesize_and_play(llm_response)

    async def classify_for_ivr(self, transcript) -> str:
        lc = transcript.lower()
        # simple heuristics; replace with model-based classifier if desired
        if any(k in lc for k in ["press", "enter", "dial", "press 1", "press 2", "keypad"]):
            return "dtmf"
        return "tts"

    def _extract_digits_or_map_intent(self, transcript) -> list[str]:
        # Example: map intent phrases to digit sequences
        # Or parse "enter 1234" -> ["1","2","3","4"]
        digits = []
        # simple parse
        for ch in transcript:
            if ch.isdigit():
                digits.append(ch)
        if digits:
            return digits
        # fallback mapping
        if "representative" in transcript or "agent" in transcript:
            return ["0"]
        return ["0"]

    async def _notify_llm_ivr_action(self, transcript, mode, data):
        # Add a short system note to session chat that an IVR action occurred
        note = f"Runtime detected IVR; executed {mode} action: {data}"
        self.session.chat_ctx.add_message(role="system", text=note)

    async def _call_llm_with_origin(self, transcript, origin):
        # Insert a short system prompt so the LLM knows this utterance is from a human
        system_note = f"Incoming utterance origin: {origin}. If origin=='human', respond naturally."
        self.session.chat_ctx.add_message(role="system", text=system_note)
        self.session.chat_ctx.add_message(role="user", text=transcript)
        # call default LLM node to get reply text
        reply = ""
        async for chunk in Agent.default.llm_node(self, self.session.chat_ctx, self.tools, self.model_settings):
            if hasattr(chunk, "text"):
                reply += chunk.text
        return reply

    async def _synthesize_and_play(self, text):
        # Before calling TTS, set a flag so tts_node knows origin
        self.session.meta = getattr(self.session, "meta", {})
        self.session.meta["utterance_origin"] = self._utterance_origin
        # call default TTS node (which we'll override to read session.meta)
        async for frame in Agent.default.tts_node(self, iter([text]), self.model_settings):
            await self.play_audio_frame(frame)

3) TTS node override (reads origin flag) • The TTS node inspects

session.meta["utterance_origin"]

(or another session field) and adjusts voice/style. If origin == "human", it also attaches a

via_human

flag in any metadata you surface.

async def tts_node(self, text_stream, model_settings):
    origin = self.session.meta.get("utterance_origin", "unknown")
    if origin == "ivr":
        # use clear, short, slower TTS suitable for IVR
        self.tts.update_options(voice="clear", speed=0.95)
    elif origin == "human":
        # human-targeted TTS; set via_human flag in output metadata
        self.tts.update_options(voice="natural", speed=1.0)
    async for frame in Agent.default.tts_node(self, text_stream, model_settings):
        # attach origin flag to audio frame metadata if needed
        frame.metadata = getattr(frame, "metadata", {})
        frame.metadata["via_human"] = (origin == "human")
        yield frame

4) LLM node notes • You can also inject the origin system message into the

llm_node

(as shown) so the LLM tailors its wording for IVR (concise, explicit DTMF instructions) vs human (conversational). • The LLM is not bypassed automatically by `ivr_detection`; your custom logic decides whether to call it or to call DTMF tools directly. 5) Edge cases and recommended mitigations • False positives (human mistaken for IVR): avoid auto-sending audible DTMF immediately. Use conservative thresholds, require IVR pattern confirmation (multiple IVR cues) or a short probe (silent frame) before DTMF. • False negatives (IVR not detected): add heuristics—if transcript contains “press X” or “enter” treat as IVR; provide a retry/fallback that attempts DTMF when stuck. • Rapid flips IVR↔️human: debounce the IVR state (require N consecutive frames/utterances to flip) and add a short suppression window after flips to avoid racey DTMF sends. • Mixed utterances: if an utterance contains both IVR cues and human speech, split into segments or prefer “ambiguous” handling—notify LLM: “ambiguous source — prefer safe behavior.” • DTMF audible leakage: only call

send_dtmf_events

when ivr detection is positive and after the runtime confirms DTMF tools are allowed. • Logging & telemetry: record ivr_state transitions, classifier decisions, and any DTMF calls to tune behavior. 6) Implementation tips • Keep the origin flag small and predictable (e.g., "ivr" or "human") and store it in session-level metadata so both llm_node and tts_node can read it. • Use an LLM-based classifier for hard cases (call a small classifier model to decide dtmf vs tts) if heuristics are insufficient. • Test flows end-to-end with recorded IVR prompts and live human calls to tune thresholds and debounce/window sizes. If you want, I can: • Produce a more complete runnable code sample for your exact SDK version with debounce logic and a small LLM-based classifier, or • Draft the exact system prompts to send to the LLM for IVR vs human behavior. Which would you like next? Sources: send_dtmf_events | LiveKit Documentation | examples/dtmf | livekit/livekit_composite | DeepWiki | GetDtmfTask | LiveKit Documentation | Agent session | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation, with interactive elements