The issue happening again. I set

turn_detection=MultilingualModel(),

and now I it sometimes doesn’t detect user turn has ended. I see multiple STT logs. That happens when when end of turn doesn’t trigger. I’m using deepgram nova 3 for stt

Thanks — I appreciate the clear detail and the urgency. You’re seeing intermittent failures where

MultilingualModel()

sometimes does not detect end-of-turn (so no LLM call), even though Deepgram Nova 3 STT produces multiple logs. I’ll help you narrow this to the root cause and suggest immediate tests and mitigations. What I’m going to check with you (short list) — please provide any of these you can: • A short, sanitized STT event sequence for a failing call (timestamps + event types like partial/final/end_of_speech). The key is to show whether Deepgram emitted any end-of-speech/final events. • A short, sanitized STT event sequence for a successful call for comparison. • The AgentSession snippet (sanitized) showing

turn_detection: MultilingualModel()

,

stt

config for Deepgram, any

vad

config, and

min_endpointing_delay

. • Client/SDK platform info for failure vs success (browser/SDK version, OS) and approximate time(s) of recent failures. Why these items matter • If Deepgram is not emitting an end-of-speech/final event the MultilingualModel may not receive the signal it expects. Even if STT logs appear, the absence of "final" endpointing signals will prevent turn detection from firing. • The MultilingualModel listens for timing and STT endpoint signals; intermittent failures can be caused by network jitter, missed RTP packets, or STT provider intermittent endpointing behavior. •

min_endpointing_delay

and any VAD settings affect how quickly the session decides a turn ended — too large or too small values can make endpointing unreliable in noisy networks. Immediate checks and quick tests to run now (one at a time) 1. Check for Deepgram “final” / end-of-speech flags: ◦ In the failing session logs, confirm whether Deepgram ever emits a final/finalized/end event. If not, that’s the immediate cause. 1. Quick A/B test: Force VAD endpointing ◦ For a short test call, set

turn_detection="vad"

with a basic

vad

config. If that reliably triggers LLM calls, the problem is in the STT → MultilingualModel signaling path. 1. Reduce endpointing latency: ◦ Temporarily set

min_endpointing_delay

to 200ms and test. This can reveal whether small timing differences are preventing endpointing. 1. Reproduce while capturing full event trace: ◦ Start a failing call and capture STT events, turn-detection events, and any agent logs (timestamps). A single trace showing STT partials but no STT final or no turn-detection event is the smoking gun. 1. Test with an alternate STT endpoint (if possible): ◦ If you can quickly test with a different STT or a simpler Deepgram config, it will help determine if Deepgram Nova 3 intermittent endpointing is the issue. What I’ll do when you share logs/config • Compare failing vs successful STT sequences to see whether Deepgram final events are missing or delayed. • Check for patterns (network region, SDK version, or time-of-day) that correlate with failures. • Recommend one of: adjust endpointing delay, temporarily use VAD for endpointing, or change STT settings (Deepgram finalization) depending on the trace. If you can paste a short sanitized STT event sequence from one failing call (and one successful call if possible) plus your AgentSession snippet, I’ll analyze and point to the most likely root cause and exact configuration change to try next.

There are close to 20 STT logs. This is the first log { “message”: “STT metrics”, “level”: “INFO”, “name”: “livekit.agents”, “model_name”: “nova-3", “model_provider”: “Deepgram”, “audio_duration”: 5.05, “pid”: 19042, “job_id”: “AJ_UeNwmmiN3Dyq”, “room_id”: “RM_XqcUMQhMgkiT”, “timestamp”: “2026-03-13T062753.894310+00:00" } and this is the last after this user disconnected the call { “message”: “STT metrics”, “level”: “INFO”, “name”: “livekit.agents”, “model_name”: “nova-3", “model_provider”: “Deepgram”, “audio_duration”: 5.05, “pid”: 19042, “job_id”: “AJ_UeNwmmiN3Dyq”, “room_id”: “RM_XqcUMQhMgkiT”, “timestamp”: “2026-03-13T062944.494651+00:00" } For almost 2 minutes STT log kept triggering. end of utterance did not trigger This is the config

In prewarm method we set proc.userdata["vad"] = silero.VAD.load()
vad = ctx.proc.userdata["vad"]
AgentSession(
    ...
		vad=vad,
		turn_detection=MultilingualModel(),
)

min_endpoint_delay is default

Thanks — this is very helpful. I hear you: Deepgram Nova-3 STT is emitting many STT metrics over ~2 minutes for the same utterance, you’re using

MultilingualModel()

for

turn_detection

,

vad

is set to a Silero VAD instance,

min_endpointing_delay

is default, and the session never triggers an end-of-turn so the LLM is not called. That points to a runtime endpointing problem rather than a config change. What this pattern means • Multiple STT logs for the same audio_duration and a long gap until disconnect indicate STT kept streaming partial results (or the session continued) without producing a final/end-of-speech signal that MultilingualModel could use to decide the turn ended. • MultilingualModel expects either explicit STT endpointing signals or timing/VAD cues to decide end-of-turn. If neither a clear STT “final” event nor a VAD transition is observed, turn detection can stall and never call the LLM. Immediate hypotheses to test (ranked by likelihood) • Deepgram occasionally does not emit an end-of-speech/final marker for that spoken segment (intermittent STT endpointing). • The MultilingualModel did not receive or missed the STT final/end events due to network jitter or event ordering. • Silero VAD did not detect speech end (or VAD output wasn't used by MultilingualModel) — when VAD and MultilingualModel disagree, endpointing can be missed. • Default

min_endpointing_delay

together with delayed signals caused the model to keep waiting and never commit an endpoint. Concrete next steps to isolate and mitigate (please run one at a time and report results) 1) Confirm STT final/end signals in raw STT events • Capture a failing call trace that includes raw Deepgram events (partial/final/end_of_speech flags) with timestamps. • If Deepgram never emits a final/end_of_speech for the utterance, that is the root cause. 2) Short A/B with forced VAD endpointing • For a quick test only, set

turn_detection="vad"

(with your Silero VAD instance still loaded) in a test AgentSession and reproduce. • If LLM calls trigger reliably, the problem is the STT → MultilingualModel endpointing path rather than VAD. 3) Temporarily lower endpointing latency • Set

min_endpointing_delay

to 200ms for a short test. This reduces the time the turn-detector waits for additional audio and can help in cases where signals arrive late. 4) Capture full trace for one failing session • Collect STT events, VAD on/off events, and MultilingualModel turn-detection events (timestamps). The smoking gun is: STT partials without a STT final, VAD never transitions to speech_end, or turn-detection never emits an end event despite one of the above. 5) Try an alternate STT finalization mode (Deepgram) • Check your Deepgram config for any settings that control finalization behavior (e.g., utterance detection, silence thresholds). If you can temporarily enable a more aggressive endpointing or change the Nova model’s silence handling, test that. 6) Short-term mitigation while we investigate • Use VAD-based turn_detection for calls where reliability is critical. • Or lower

min_endpointing_delay

so the agent commits earlier when ambiguous. What I can do for you next • If you can paste one sanitized failing trace showing the sequence of Deepgram events (include partial/final labels and timestamps), plus any VAD on/off logs for the same timeframe, I will pinpoint whether Deepgram is failing to finalise or whether the MultilingualModel/VAD timing is the issue. • If you prefer, paste the Deepgram event sample and I’ll mark the exact moment the endpoint should have happened and why it didn’t. I know this is high-impact — once you share one short failing trace (STT events + VAD events if available), I’ll analyze it and give an exact targeted fix to stop missed endpoints.