My multimodal realtime agent is behaving as if the user spoke immediately after it, and cutting off actual user speech with a response to an empty string. How can I prevent this?

Thanks for reaching out, Matthew. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I am using semantic vad: session = AgentSession( min_interruption_duration=1.0, vad=vad, llm=openai.realtime.RealtimeModel( turn_detection=TurnDetection( type="semantic_vad", eagerness="low", create_response=True, interrupt_response=False, ), ), ) I also have interruptions off: @session.on("speech_created") def _on_speech_created(ev): ev.speech_handle.allow_interruptions = False sentry_sdk.add_breadcrumb(category="tts", message="speech_created", level="info")

Thanks — that’s very helpful. You’re already using semantic VAD with low eagerness and disallowing interruptions, so you’ve taken good first steps. Because the agent is still responding to empty input, let’s focus on three targeted options you can try (from least to most impactful). I’ll also ask one quick clarifying question at the end to fine-tune recommendations. Quick summary of what I’m responding to: • You’re using semantic VAD with

eagerness="low"

,

create_response=True

,

interrupt_response=False

,

min_interruption_duration=1.0

, and you set

allow_interruptions = False

in

speech_created

. Options to try 1) Add a false-interruption timeout / resume behavior • Semantic VAD can still produce false interruptions. Add a short delay before treating an interruption as a real user turn so the agent won’t immediately respond to silence. • If your session supports

false_interruption_timeout

and

resume_false_interruption

, set a small timeout (start with 1–2 seconds) and enable resume so the agent will continue speaking if no user speech follows. • This prevents an immediate response when the VAD fires but no transcript arrives. 2) Tighten your transcript-level guard in your handlers (defensive check) • Ensure you ignore empty final transcripts before creating a response. Add a guard like:

@session.on("user_input_transcribed")
def on_transcript(ev):
    if not ev.transcript.is_final:
        return
    if not ev.transcript.text.strip():
        # empty final transcript — ignore
        return
    # proceed with processing real user input

• This ensures downstream logic never builds a reply from an empty string even if the VAD signals a turn. 3) Consider switching to Server VAD or adjust semantic VAD timing • Server VAD (if available) lets you increase

silence_duration_ms

and adjust a numeric threshold, which is often more conservative than semantic VAD for noisy environments. • If you prefer to remain on semantic VAD, re-check any timing controls you can increase (e.g., lengthening any silence window or increasing

min_interruption_duration

further, if configurable). Try bumping

min_interruption_duration

from 1.0 to 1.5–2.0s temporarily to see if it reduces false triggers. Why these help • A timeout + resume prevents acting on transient false positives. • A transcript guard prevents empty-string responses from application logic. • Server VAD or longer interruption durations reduce the chance the system misclassifies breaths or background noise as completed user speech. One quick question so I can give exact code you can paste in: • Does your AgentSession implementation already accept

false_interruption_timeout

and

resume_false_interruption

parameters (or similar names)? If so, tell me the exact parameter names your SDK exposes. If it does not, I’ll provide the guard + semantic tweaks you can apply now and a safe Server VAD snippet as an alternative. If you want, paste the transcript event payload for one of the false-interruption occurrences (or the exact SDK version you’re using) and I’ll suggest the minimal changes with concrete values to try next.

My agent cannot be interrupted at all so I am not sure if false interruption detection is even an issue. The interaction should be fully turn-based. I do have this: @session.on("user_input_transcribed") def _on_user_input_transcribed(ev: UserInputTranscribedEvent): if not ev.is_final: return logger.info( f"[USER STT] ({ev.language}, final={ev.is_final}) {ev.transcript!r}" )

Thanks — this is very clear and helpful. You’re already doing the right things (semantic VAD, low eagerness, disallowing interruptions, and listening for

user_input_transcribed

). The remaining cause is that the STT/turn-detection pipeline can produce a final transcription that’s empty (silence, noise, or non-verbal audio). When that final empty transcript reaches your logic, the agent can still create a response. What to do now — minimal, safe changes you can apply immediately 1) Defensively ignore empty final transcripts • You already check

ev.is_final

. Add one more check for empty (or whitespace-only) transcript text before you log/process it. Example (drop into your current handler):

@session.on("user_input_transcribed")
def _on_user_input_transcribed(ev: UserInputTranscribedEvent):
    if not ev.is_final:
        return

    transcript = (ev.transcript or "").strip()
    if not transcript:
        <http://logger.info|logger.info>("Empty or silent user input detected — ignoring final transcript.")
        return

    <http://logger.info|logger.info>(f"[USER STT] ({ev.language}, final={ev.is_final}) {transcript!r}")
    # continue processing real user input...

Why: This ensures downstream code never builds a reply from an empty string even if turn detection produces a final event. 2) Ensure the agent doesn’t auto-create a response to empty input • You have

create_response=True

in

TurnDetection

. If the SDK or agent layer will automatically create a response when a final turn is detected, keep

create_response=True

only if you’re certain your event handler will block empty transcripts. Otherwise, consider setting

create_response=False

and invoking response creation explicitly when you confirm a non-empty final transcript. 3) (Optional) Slightly increase conservative timing if false empty finals persist • If you still see empty finals, raise

min_interruption_duration

slightly (e.g., 1.5–2.0s) so the turn detector waits longer before emitting finalization. This is conservative and safe in turn-based flows. • If you can switch to a server-configurable VAD later, Server VAD lets you tune

silence_duration_ms

and threshold numerically — useful in noisy environments. Why this will solve your issue • With interruptions disabled, false interruption events are unlikely to be the culprit. The remaining path is an empty final transcript reaching your processing logic. The empty-transcript guard guarantees the agent only reacts to real user speech. Adjusting

create_response

or the finalization timing gives you additional control if you prefer the agent to only respond when you explicitly tell it to. If you want, I can: • Provide a snippet to explicitly call whatever function creates the agent response only after your non-empty final check, or • Suggest conservative numeric defaults for

min_interruption_duration

or

create_response

behavior tailored to your latency/UX tradeoffs. Which would you prefer? Sources: Agent speech and audio | LiveKit Documentation | OpenAI Realtime API plugin guide | LiveKit Documentation

This started happening after I added a tool that called a separate text based ai to summarise sessions. Could this be a cause?

Thanks — that context is very helpful. Yes, the summarization tool is a likely cause. Why the summarization tool can trigger the behavior • If your tool runs on every final transcription event, it will receive empty final transcripts (silence/noise) just like the rest of your pipeline. • If that tool sends the empty text to a text-only AI and then returns or triggers a response back into your agent (or if

create_response=True

lets the agent auto-generate a reply after the tool runs), the agent can end up replying to an empty string. • In short: an automated tool invoked on every final transcript can convert an empty final STT result into a real agent response. Concrete fixes to try (safe, minimal changes) 1) Guard the tool invocation with a non-empty final-check • Ensure you only call the summarization tool when

ev.is_final

is true AND the transcript contains non-whitespace text. Example (integrate into your existing handler):

@session.on("user_input_transcribed")
def _on_user_input_transcribed(ev: UserInputTranscribedEvent):
    if not ev.is_final:
        return

    transcript = (ev.transcript or "").strip()
    if not transcript:
        <http://logger.info|logger.info>("Empty or silent user input detected — skipping summarization and response.")
        return

    # Now it's safe to call the summarization tool
    summary = call_session_summarizer(transcript)
    # Process summary / send to agent as needed

2) Prevent automatic agent responses when tools run • If

create_response=True

or your tool flow implicitly triggers the agent to reply, consider: ◦ Setting

create_response=False

in turn detection and explicitly creating a response only after your non-empty checks and tool results. ◦ Or keep

create_response=True

but ensure the tool or downstream code does not feed empty text back into the agent. 3) Add defensive logic inside the tool pathway • Inside the summarization tool, ignore empty inputs before calling the text-based AI. • If the tool produces output that is routed back to the agent, also check that the tool output is non-empty and intended to be a user-facing response before injecting it. 4) Add logging to trace the flow • Log when you receive final transcripts, when you call the summarizer, and when the summarizer returns. That will show whether the tool ran on an empty transcript and whether it caused the agent to reply. What I can help with next • If you paste the snippet where you call the summarizer (or how the summarizer returns results into the agent), I’ll suggest the exact guard and show how to prevent the tool from triggering a reply on empty input. • If you want, I can also show the pattern to explicitly create agent responses only after you validate the summarizer output (so automatic replies are disabled unless you choose). Which snippet would you like me to review — the place you call the summarizer, or the code that injects the summarizer’s output back into the agent?

Documentation suggests that input gating instead of disabling interruptions is the best solution. Can you elucidate?

Thanks — this is a great question. You’ve given the right intuition: disabling interruptions and input gating are different tools and input gating is the better fit for preventing responses to empty/silent input while keeping a natural, turn-based UX. Below I’ll explain the difference, why input gating solves your issue (especially after adding your summarizer tool), and give concise, practical changes you can apply right away. What input gating is (short) • Input gating means validating/filtering a user’s turn before letting it trigger any LLM/tool calls or agent responses. • It inspects the final transcription (and optionally metadata like confidence, word count, semantic checks) and only “opens the gate” when the input meets your criteria. Why this is better than just disabling interruptions • Disabling interruptions (

allow_interruptions=False

) only prevents the user from cutting the agent off while it speaks. It does not prevent the agent from responding to an empty final transcript once the agent finishes speaking. • Input gating prevents the agent and any downstream tools (like your summarizer) from being invoked on empty/noise transcripts — the exact problem you’re seeing after adding the summarizer. • Input gating preserves natural interactivity: users can still interrupt when appropriate, and you don’t need to globally block input during playback. How the summarization tool likely caused the problem • Your summarizer runs on final transcription events. If a final transcript is empty (silence/noise), the tool still runs, returns something (or returns empty), and that output may be routed back to the agent — causing a reply to an empty string. • If the agent auto-creates responses on turn completion (

create_response=True

), the flow can convert empty STT -> tool call -> agent response. Minimal, practical fixes (apply now) 1) Gate at transcription: only call tools / trigger responses on non-empty final transcripts • Replace your current handler with this guarded pattern:

@session.on("user_input_transcribed")
def _on_user_input_transcribed(ev: UserInputTranscribedEvent):
    if not ev.is_final:
        return

    transcript = (ev.transcript or "").strip()
    if not transcript:
        <http://logger.info|logger.info>("Empty or silent user input detected — skipping summarizer and agent response.")
        return

    # safe to call summarizer and trigger agent response
    summary = call_session_summarizer(transcript)
    # continue with normal processing

• This prevents your summarizer from being called on empty input and blocks any automatic responses that follow. 2) Gate in tool code (defensive programming) • Inside the summarization tool, also ignore empty inputs:

def call_session_summarizer(text):
    text = (text or "").strip()
    if not text:
        return None  # or raise/return sentinel to avoid injecting content back to agent
    # call text-AI

• This ensures the tool never feeds empty text back into the agent pipeline. 3) Ensure your turn/response creation is explicit • If

create_response=True

in your

TurnDetection

, the agent may auto-create responses on turn completion. Consider: ◦ Keep

create_response=True

but ensure your handlers block empty transcripts (as above), OR ◦ Set

create_response=False

and explicitly call the agent response creation only after gating and (optionally) after tool results are confirmed non-empty. 4) Add a gate in lifecycle hooks for LLM generation • If you have an

on_user_turn_completed

hook or equivalent, inspect

new_message.text_content

and raise StopResponse() if it’s empty or too short:

async def on_user_turn_completed(self, turn_ctx, new_message):
    text = (new_message.text_content or "").strip()
    if not text or len(text.split()) < 2:  # tune threshold as needed
        raise StopResponse()

• This prevents the LLM and tools from being invoked on low-quality input. Tuning and additional checks (optional) • Word-count threshold: require >= 2–3 words for a turn to be valid. • Confidence threshold: if your STT provides confidence, skip low-confidence finals. • Simple semantic check: run a fast intent or keyword check if you need to allow single-word commands. Suggested immediate defaults to try • Transcript gate: require non-empty, and optionally len(words) >= 2. • Summarizer: do not run on empty input; return early. • If you keep

create_response=True

, apply the transcript gate strictly — else set

create_response=False

and create responses explicitly after validation. How this resolves your situation • Input gating stops the summarizer from receiving/processing empty finals, which is the likely causal chain that turned silence into an agent response. • It prevents automatic replies without forcing you to disable interruptions globally, maintaining better UX. If you’d like, I can: • Review the exact snippet where the summarizer is invoked and show the minimal place to add the guard, or • Show how to change your

create_response

usage safely if you prefer explicit response creation. Which would you prefer me to review (summarizer call site or agent response creation code)? Sources: LLM-Powered Content Filter | LiveKit Documentation | Interrupt User | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation | Turns overview | LiveKit Documentation | Text and transcriptions | LiveKit Documentation