I am seeing a strange issue a stt transcript is not be returned at all. But we see the stt metric running in the logs.

Thanks for reaching out, Andy. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Issue: STT (Deepgram nova-3) fails to produce transcripts for short utterances on inbound SIP calls We're seeing a recurring issue where the STT pipeline detects speech (VAD fires, STT metrics are collected) but never produces a transcript for short consumer responses on inbound SIP calls. This results in the bot going silent and the consumer eventually hanging up. Setup: - livekit-agents~=1.2 - livekit-plugins-noise-cancellation~=0.2 - STT: Deepgram nova-3, language="multi", interim_results=False, endpointing=500 - VAD: Silero, activation_threshold=0.35 - Turn detection: MultilingualModel() - Noise cancellation: BVCTelephony() for SIP participants, BVC() for non-SIP - min_endpointing_delay=0.5, max_endpointing_delay=3.0 What we observe: Across multiple inbound SIP calls (all on LiveKit Cloud, prod), the consumer speaks short responses like "Yes", "Later", or "Yeah what's up." We see: 1. VAD detects speech (consumer_speech_detected = True) 2. Multiple STT metrics are logged (5-7 per utterance, with audio_duration values ranging from 0.7s to 5s) 3. No transcript is ever produced — no user_input_transcribed event fires 4. No EOU event fires 5. The bot sits in silence waiting for a response 6. Consumer eventually hangs up Longer utterances on the same calls (e.g., "Hello?" after waiting) DO get transcribed successfully. The issue seems specific to short initial responses on inbound SIP calls. Outbound calls with the same configuration work fine. Example (EventLogID 94975223): - 231254 — Bot speaks greeting: "Am I speaking with Sharfuddin?" - 231282 — VAD detects consumer speech (consumer says "Yes") - 231283 → 231289 — 5 STT metrics fire (audio_duration: 4.05, 0.7, 5, 0.95, 4.8s) with NO transcript - 231314 — Consumer says "Hello?" (checking if anyone is there) — this one IS transcribed - 231323 — Consumer hangs up Two other calls with the same pattern: - EventLogID 94952280: Consumer says "Yeah what's up" → 7 STT metrics, no transcript, 25s silence, hangup - EventLogID 94955339: Consumer says "Later" → STT metrics, no transcript, 9s silence, hangup Questions: 1. Could BVCTelephony() noise cancellation be too aggressive for short/quiet speech on SIP audio? Would disabling it or switching to BVC() for testing help isolate this? 2. Could language="multi" cause issues with short English words vs using "en"? 3. Is there a known interaction between interim_results=False + endpointing=500 that could cause short utterances to be swallowed? 4. What do the STT metrics represent — per-transcript or per-audio-chunk? If STT metrics fire but no transcript is produced, does that mean Deepgram received audio but returned empty results?

Thanks — this is a clear, helpful report. I’ll respond to each of your four questions directly, summarize what the STT metrics imply, and give a prioritized troubleshooting plan you can run quickly on prod or a dedicated test call. Quick acknowledgement • You’re seeing VAD fire and STT metrics (so audio reaches the STT pipeline) but no

user_input_transcribed

or EOU events for short inbound SIP replies. Longer replies transcribe fine and outbound calls behave normally. I’ll focus on what can cause short utterances to be dropped and the exact checks to run. Answers to your questions 1) Could BVCTelephony() be too aggressive for short/quiet speech? • Yes. BVCTelephony() is optimized for telephony (narrowband/SIP) and is more aggressive than the full-band BVC() filter. Aggressive NC can suppress short/quiet speech, especially on degraded or already pre-processed SIP audio. • Double-processing (e.g., BVCTelephony() plus krisp on the SIP trunk) is discouraged — pre-processed audio can remove initial words or reduce confidence. • Action to test: temporarily disable noise cancellation for SIP participants (or switch to BVC() or no NC) and re-run the failing flow. If short utterances start appearing, BVCTelephony() is the likely culprit. 2) Could language="multi" cause issues with short English words vs "en"? • Yes. Using

multi

adds language-detection complexity. Very short utterances (1–3 words) give little context for language ID and can increase uncertainty or lower confidence, which may lead to no final transcript. • Action to test: set Deepgram to

language="en"

for these calls and see if short English replies are transcribed. If that fixes it, use

en

for English-only traffic or route SIP calls with known language to language-specific models. 3) Is there an interaction between interim_results=False + endpointing=500 that swallows short utterances? • Potentially. With

interim_results=False

the system will not emit intermediate partial transcripts. If STT or VAD fails to produce a confident final result for a very short utterance, nothing will be emitted. •

endpointing=500

(500 ms) sets a short end-of-utterance window; combined with tight VAD settings, it can be easy to miss brief utterances that have minimal pre-speech or post-speech silence. • Action to test: ◦ Enable interim results (set

interim_results=True

) and observe whether any interim transcripts appear. If you see interim text but no final, it indicates the model is unsure/confidence thresholds stop finalization. ◦ Slightly relax endpointing/VAD timing (increase

min_speech_duration

or increase endpointing slightly) to capture more context around short replies. 4) What do the STT metrics represent and what does it mean when metrics fire but no transcript is produced? • STT metrics are emitted per STT request/audio chunk processed (they include fields like

audio_duration

,

duration

,

streamed

,

speech_id

). • Seeing STT metrics but no

user_input_transcribed

or EOU means: audio reached and was processed by STT, but STT did not return a confident, final transcript for that request (it might have returned empty/low-confidence results or nothing final). • Practical interpretation: STT received audio (so transport/VAD forwarded it), but the STT model either: ◦ returned no text (empty result), ◦ returned only very low-confidence text which the pipeline suppresses, or ◦ produced only interim hypotheses that never finalized. • Action: log full STT metric payloads including

speech_id

and any confidence fields. Compare

audio_duration

to the expected spoken duration—if

audio_duration

is large for a single-word reply, you may have buffering or late VAD start. Prioritized troubleshooting checklist (run in this order) 1) Short, fast isolating tests • On a test inbound SIP call, disable noise cancellation (BVCTelephony) for that participant and run the same greeting flow. Observe whether

user_input_transcribed

fires for the short reply. • Also disable any krisp on the SIP trunk if enabled. 2) Language test • Switch Deepgram to

language="en"

for this test group and confirm whether short English replies transcribe reliably. 3) Observe interim results • Enable

interim_results=True

(use the Deepgram plugin config if required) and watch for interim transcripts. If you get interim but no final, that points to confidence/endpointing/finalization behavior. 4) Relax VAD/endpointing slightly • Increase endpointing or VAD timeouts a bit (e.g., endpointing -> 700–800ms, loosen activation_threshold) and retry. This gives more pre/post context for short utterances. 5) Log/compare STT metrics • Capture and share (for multiple failed calls) the full STT metrics payloads:

speech_id

,

audio_duration

,

duration

, and any confidence fields. Look for patterns like multiple long

audio_duration

entries for a short utterance (suggests buffering) or consistently low processing duration vs audio duration. 6) Isolate STT from downstream pipeline • Run an STT-only test agent that immediately logs any transcripts (interim + final). This rules out downstream LLM/pipeline logic swallowing results. What I’d like you to send next (so I can give exact config/code changes) • One example STT metrics payload for a failed short utterance (include

speech_id

,

audio_duration

,

duration

, timestamps). • Whether krisp (or other NC) is enabled on your SIP trunk. • The exact Deepgram plugin config snippet you’re using (so I can confirm where to toggle

interim_results

or

language

). • A minimal snippet showing how you subscribe to the SIP participant mic and your

user_input_transcribed

handler (so I can confirm nothing in event handling is inadvertently filtering). If you prefer, start with steps 1–3 (disable BVCTelephony, set language to

en

, enable interim results) and tell me the outcome — I’ll interpret the STT metrics you paste and recommend the final configuration to prevent regression. Sources: Choosing the right noise cancellation for your agent | LiveKit | Troubleshooting STT not picking up utterances | LiveKit | Deepgram STT | LiveKit Documentation | Deepgram STT plugin guide | LiveKit Documentation | STT Metrics | LiveKit Documentation

Answers to your follow-up questions: 1. STT metrics payload for a failed short utterance Here are the STT metrics from EventLogID 94975223 — consumer said "Yes" after the bot's greeting. No transcript was ever produced for the "Yes". The first transcript that came through was "Hello?" ~22 seconds later. The default metrics.log_metrics() only extracts audio_duration from STTMetrics. Here are the raw log entries for the failed "Yes" utterance (timestamps are Unix ms): ┌─────┬──────────────────────────┬────────────────┬─────────────────┬───────────────────────────────────────────────┐ │ # │ timestamp │ audio_duration │ model_name │ notes │ ├─────┼──────────────────────────┼────────────────┼─────────────────┼───────────────────────────────────────────────┤ │ 1 │ 1771024374416 (231254) │ 2.1s │ deepgram/nova-3 │ During bot greeting playout │ ├─────┼──────────────────────────┼────────────────┼─────────────────┼───────────────────────────────────────────────┤ │ 2 │ 1771024379249 (231259) │ 4.8s │ deepgram/nova-3 │ Bot greeting finished ~231300 │ ├─────┼──────────────────────────┼────────────────┼─────────────────┼───────────────────────────────────────────────┤ │ 3 │ 1771024383283 (231303) │ 4.05s │ deepgram/nova-3 │ Consumer said "Yes" ~231303 │ ├─────┼──────────────────────────┼────────────────┼─────────────────┼───────────────────────────────────────────────┤ │ 4 │ 1771024383981 (231303) │ 0.7s │ deepgram/nova-3 │ Same speech region │ ├─────┼──────────────────────────┼────────────────┼─────────────────┼───────────────────────────────────────────────┤ │ 5 │ 1771024388982 (231308) │ 5.0s │ deepgram/nova-3 │ Silence after "Yes" │ ├─────┼──────────────────────────┼────────────────┼─────────────────┼───────────────────────────────────────────────┤ │ 6 │ 1771024389928 (231309) │ 0.95s │ deepgram/nova-3 │ Still no transcript │ ├─────┼──────────────────────────┼────────────────┼─────────────────┼───────────────────────────────────────────────┤ │ 7 │ 1771024394736 (231314) │ 4.8s │ deepgram/nova-3 │ "Hello?" — this one DOES produce a transcript │ └─────┴──────────────────────────┴────────────────┴─────────────────┴───────────────────────────────────────────────┘ VAD detected speech at 231302.955 (consumer_speech_detected = True). The first user_input_transcribed event fires at 231314.736 with text "Hello?". No transcript was produced for the "Yes" spoken ~12 seconds earlier. Note: STTMetrics in livekit-agents v1.2 does not include speech_id or confidence — only type, label, request_id, timestamp, duration, audio_duration, streamed, and metadata. We can add custom logging to capture request_id and duration if that would help. 2. Krisp / NC on SIP trunk No krisp or noise cancellation is enabled on the SIP trunk itself. The trunk config is minimal: { "sipTrunkId": "ST_qXRiWtXWSvPn", "allowedAddresses": ["0.0.0.0/0"], "headersToAttributes": { ... } } The only noise cancellation is BVCTelephony() applied in the agent code via room_options. No double-processing. 3. Deepgram plugin config We're using the LiveKit inference API (not the Deepgram plugin directly): stt=inference.STT( model="deepgram/nova-3", language="multi", extra_kwargs={ "interim_results": False, "filler_words": False, "endpointing": 500, "punctuate": True, "smart_format": True, "numerals": True, }, ), With: vad=silero.VAD.load(activation_threshold=0.35) turn_detection=MultilingualModel() min_endpointing_delay=0.5 max_endpointing_delay=3.0 4. Subscription and user_input_transcribed handler We don't manually subscribe to tracks — session.start() handles subscription automatically: await session.start( room=ctx.room, agent=agent, room_options=room_io.RoomOptions( audio_input=room_io.AudioInputOptions( noise_cancellation=lambda params: ( noise_cancellation.BVCTelephony() if params.participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP else noise_cancellation.BVC() ), ), ), ) Our user_input_transcribed handler: @session.on("user_input_transcribed") def _on_user_transcript(ev): cancel_greeting_timer("transcript received") cancel_dead_air_timer("transcript received") if not agent.amd_handler.first_transcript_received.is_set(): transcript = ev.transcript if hasattr(ev, 'transcript') else str(ev) logger.info(f"First transcript received: '{transcript}'") agent.amd_handler.set_first_transcript(transcript) if hasattr(agent, 'call_state') and not agent.call_state.state.consumer_spoke: agent.call_state.set("consumer_spoke", True) if hasattr(agent, 'call_state') and agent.call_state.state.awaiting_voicemail_beep: agent.amd_handler.reset_voicemail_timeout() Nothing in this handler filters or drops transcripts — it only reacts to the first one and doesn't prevent subsequent transcripts from flowing through the pipeline.

Thanks — this is an excellent, detailed report. I can see VAD fired and multiple STT metric entries for the short “Yes” reply, but no transcript or EOU, while a later longer “Hello?” is transcribed. That rules out basic audio transport and subscription problems and focuses us on STT/processing behavior. Here’s a concise analysis of what the logs imply, followed by targeted, ordered tests and exact config/code changes to run next. What the STT metrics tell us • Metrics show Deepgram was invoked multiple times around the short reply (audio_duration entries: 2.1s, 4.8s, 4.05s, 0.7s, 5.0s, 0.95s). That means audio chunks reached the STT pipeline repeatedly. • No

user_input_transcribed

or EOU fired for that utterance → Deepgram either returned no confident final result, returned only interim hypotheses that never finalized, or the pipeline suppressed low-confidence finals. • The presence of many and sometimes long audio_duration values for a single-word reply suggests VAD/audio-chunk alignment or buffering: VAD may be grouping non-contiguous audio or silence with speech, producing multiple requests that still don’t finalize. • Because you’re using

interim_results=False

and endpointing=500, if the model hesitates to finalize on short inputs you’ll see no emitted transcript even though Deepgram processed audio. Prioritized tests to run (in this order) 1) Quick NC (noise cancellation) test — highest probability hit • Temporarily disable BVCTelephony() for SIP participants (use no NC or BVC()). BVCTelephony is aggressive for narrowband SIP and can suppress short words. • Change your RoomOptions audio filter to return None or BVC() for SIP participants, then reproduce the exact greeting → short-reply flow. • Expected: If transcripts start appearing for “Yes/Later”, BVCTelephony was the cause. Config example (temporary change)

await session.start(
  room=ctx.room,
  agent=agent,
  room_options=room_io.RoomOptions(
    audio_input=room_io.AudioInputOptions(
      noise_cancellation=lambda params: None  # temporarily disable for SIP
    ),
  ),
)

2) Language test — remove language detection ambiguity • Switch Deepgram model language from

"multi"

to

"en"

for this test cohort:

stt=inference.STT(
  model="deepgram/nova-3",
  language="en",
  extra_kwargs={
    "interim_results": False,
    "endpointing": 500,
    ...
  }
)

• Expected: If short English replies are transcribed with

en

, language detection was lowering confidence for very short inputs. 3) Enable interim results to observe in-flight hypotheses • Temporarily enable interim results to see if Deepgram produces partial text that never finalizes:

stt=inference.STT(
  model="deepgram/nova-3",
  language="en",
  extra_kwargs={
    "interim_results": True,
    "endpointing": 500,
    ...
  }
)

• Watch

user_input_transcribed

events for interim text. If you see interim but no final, the model is delivering hypotheses but not committing them. 4) Slightly relax endpointing/VAD timing • Increase endpointing to 700–800 ms and/or loosen Silero activation threshold slightly (e.g., from 0.35 → 0.3) so short words have more pre/post context: ◦ endpointing: 700 ◦ vad activation_threshold: 0.30 ◦ min_endpointing_delay: 0.7 • Rationale: gives VAD and STT more context, allows short replies to be captured and finalized. 5) Add more detailed STT request logging (request_id and duration) • Because current STTMetrics lack `speech_id`/confidence, capture request_id and duration so you can correlate requests to Deepgram and spot returned empties: ◦ Log

request_id

,

duration

, and any

metadata

fields available. If your library surfaces the raw response metadata, log it. • Example: extend your metrics logging to include

request_id

and

duration

. If needed, add a thin wrapper around the STT call to capture raw request/response. 6) Run an STT-only minimal agent • Create a minimal session that logs every interim and final transcript produced (no downstream LLM). This rules out downstream pipeline suppression. • If STT-only agent logs nothing for these short utterances, the problem is upstream (VAD/NC/STT). If it logs text but your agent still doesn’t react, look at handler timing/filters. Additional observations & why these steps are prioritized • BVCTelephony is the most likely cause because it’s optimized/aggressive for telephony audio and you’re seeing failures only on inbound SIP calls (not on outbound). You also confirmed no krisp on the trunk, so NC only exists on the agent side. •

language="multi"

plus very short utterances is the second most likely contributor: short tokens provide little evidence for language ID and can reduce finalization confidence. •

interim_results=False

hides partial hypotheses which would help confirm whether the model is attempting to transcribe. • The odd audio_durations (longer values for a short “Yes”) hint at buffering, misaligned VAD windows, or multiple chunks being built up by the VAD/turn detector before STT finalizes. What to capture and paste if tests don’t resolve it • One failed call’s full STT metrics entries (raw JSON) including request_id and duration if you can add them. • Any interim transcripts you observe after enabling interim_results. • The exact small change you made (NC disabled vs BVC vs language=en) and the result for the failing greeting flow. • If you still see repeated long audio_duration values for a one-word reply, paste a short excerpt of the VAD timestamps around the utterance (VAD start/stop) and the STT metric timestamps — that helps confirm misalignment. If you want, I can provide: • The minimal STT-only agent snippet to paste into your environment. • A small patch to add request_id / duration logging in livekit-agents v1.2 (if you confirm you want that). Next step recommendation • Start with disabling BVCTelephony for SIP participants and enable

interim_results=True

+ language="en". Run 5–10 test inbound calls with the same greeting script and paste one failing call’s raw STT metrics (including request_id and durations) if transcripts still don’t appear. I’ll analyze the timestamps and metric patterns and tell you precisely which component (VAD, NC, Deepgram, or buffering) is causing suppression and the config to permanently fix it. If you’d like, run step 1 now and tell me the result — I’ll interpret the metric differences and give the next exact config to apply in production.

Could it be my version: "livekit-agents[silero,turn-detector]~=1.2", "livekit-plugins-noise-cancellation~=0.2",

Thanks — that version info is helpful. Short answer • Yes, your current packages can contribute to the issue. Staying on livekit-agents~=1.2 and livekit-plugins-noise-cancellation~=0.2 increases the likelihood of missing short SIP utterances. Upgrading and/or testing with NC disabled is recommended. Why (brief) • The noise-cancellation plugin version you’re using applies BVCTelephony(), which is aggressive for narrowband SIP and can suppress short/quiet speech. You’re seeing the classic symptom: VAD and STT requests happen, but no final transcript for 1–3 word replies. • livekit-agents versions before 1.3 lack some VAD/turn-detection and STT-finalization refinements added in later releases that improve detection of short utterances and the coordination between VAD, endpointing, and STT. What I recommend you do next (order matters) 1. Quick isolate (highest chance to reveal cause) ◦ Temporarily disable noise cancellation for SIP participants (or switch to no NC/BVC()) and reproduce the failing greeting flow. ◦ If transcripts appear for “Yes”/“Later”, BVCTelephony() is the cause. 1. Short-term config tests (if NC disable doesn’t fully resolve) ◦ Change Deepgram language to

"en"

for these SIP calls (instead of

"multi"

). ◦ Enable interim results (

interim_results=True

) to observe partial hypotheses. ◦ Slightly relax endpointing (e.g., 700–800 ms) and/or lower VAD activation threshold a little. 1. Upgrade (recommended) ◦ Upgrade livekit-agents to >=1.3.x and the noise-cancellation plugin to the latest compatible release. These releases include improvements to VAD sensitivity, endpointing behavior, and turn detection that help with short utterances. 1. Add logging to confirm behavior ◦ Add request-level logging (request_id/duration) around STT requests so you can correlate metrics to transcripts and confirm whether Deepgram returns empty/low-confidence results. ◦ If you enable interim results, capture those outputs for failed calls. If you want, I can: • Provide the exact minimal change snippets for disabling BVCTelephony, toggling interim_results, and switching language to

en

. • Provide the small STT-only agent snippet to isolate STT output. • Suggest a safe upgrade command and pinpoint which versions to target. Which would you like to try first — disabling BVCTelephony for SIP tests, or I can give the exact code snippets to enable interim results and set language to

en

? Sources: Using Noise Cancellation for SIP and WebRTC Participants | Troubleshooting STT not picking up utterances | LiveKit | Agents (Python) | livekit/livekit_composite | DeepWiki | examples/voice_agents | livekit/livekit_composite | DeepWiki | Choosing the right noise cancellation for your agent | LiveKit

So is this not an issue that has been reported before with deepgram? INFO 184724.365 [89772445] STT metrics INFO 184727.476 [89772445] STT metrics INFO 184732.485 [89772445] STT metrics INFO 184735.600 [89772445] STT metrics INFO 184739.513 [89772445] STT metrics INFO 184742.648 [89772445] STT metrics INFO 184745.755 [89772445] STT metrics INFO 184749.809 [89772445] STT metrics INFO 184753.075 [89772445] STT metrics

Yes — this behavior is known and expected in many real-time STT setups, including Deepgram in live conversational pipelines. Why this happens (short) • Very short utterances (1–3 words) often don’t provide enough audio context for STT models to confidently segment and finalize a transcript. STT providers (Deepgram included) may process audio (you’ll see STT metrics) but return no final transcript if confidence or endpointing conditions aren’t met. • Aggressive noise cancellation or narrowband SIP preprocessing can further reduce intelligible audio for brief replies, making the problem worse even though metrics still show activity. Why your logs match that pattern • You see repeated STT metrics around the short “Yes” but no

user_input_transcribed

or EOU. That means audio reached the STT pipeline and was processed, but no confident final result was produced — consistent with short-utterance behavior and endpointing/finalization rules. Recommended mitigations (ordered by likely impact) 1. Disable/loosen noise cancellation for SIP participants (quick isolate) ◦ BVCTelephony() is aggressive for narrowband SIP and frequently suppresses short speech. Temporarily disable it (or use no NC/BVC()) and retest. 1. Switch Deepgram language to "en" for English calls ◦

language="multi"

adds language-detection uncertainty for very short replies. Use

language="en"

for English traffic to improve confidence. 1. Enable interim results to observe in-flight hypotheses ◦ Set

interim_results=True

temporarily to see whether the model is generating partial text that never finalizes. That helps distinguish confidence vs endpointing problems. 1. Relax VAD/endpointing slightly ◦ Increase endpointing (e.g., 700–800 ms) or reduce Silero activation threshold so brief words have more pre/post context and are more likely to finalize. 1. Add request-level logging for STT requests ◦ Capture request_id/duration and any returned metadata so you can correlate metric entries with responses (to confirm empty/low-confidence results). 1. Upgrade versions if feasible ◦ Later livekit-agents/noise-cancellation releases include improvements to VAD sensitivity and turn-detection behavior; upgrading reduces the chance of these edge cases. If you want, I can: • Give exact code snippets to disable BVCTelephony, toggle interim results, and switch to

language="en"

. • Provide a minimal STT-only test agent you can run to isolate STT behavior. References that explain this behavior and the tweaks above: • Troubleshooting STT not picking up utterances (guidance on short utterances and NC): https://livekit.io/field-guides/guide/troubleshooting-stt-not-registering-user-speech-sip-calls • STT metrics and what they mean: https://docs.livekit.io/recipes/metrics_stt • VAD and turn-detection tuning: https://livekit.io/field-guides/guide/vad-turn-detection-configuration • Deepgram STT plugin details: https://docs.livekit.io/agents/models/stt/plugins/deepgram Which quick test would you like to run first — disable BVCTelephony for SIP calls, or enable interim_results + language="en"? I’ll provide the code snippet for whichever you pick. Sources: Troubleshooting STT not picking up utterances | LiveKit | STT Metrics | LiveKit Documentation | VAD and turn detection configuration guide | LiveKit | Deepgram STT plugin guide | LiveKit Documentation | Improving voice AI's turn detection with transformers

How can I report to issue to livekit team?

Thanks — I can help you report this so the LiveKit team has everything they need to investigate quickly. What to include in your report (copy/paste checklist) • Short title: one-line summary ◦ Example: “STT (Deepgram nova-3) logs metrics but returns no transcripts for short inbound SIP utterances” • Environment ◦ livekit-agents version:

livekit-agents[silero,turn-detector]~=1.2

◦ noise cancellation plugin version:

livekit-plugins-noise-cancellation~=0.2

◦ LiveKit hosting: LiveKit Cloud (prod) • Reproduction steps (minimal, deterministic steps) a. Make an inbound SIP call to the bot running the above packages. b. Bot plays greeting: e.g., “Am I speaking with X?” c. Consumer responds with a short reply (1–3 words) e.g., “Yes”, “Later”. d. Observe VAD fires and STT metrics logged, but no

user_input_transcribed

or EOU event; bot remains silent. • Exact STT configuration ◦ STT: Deepgram nova-3 ◦ STT settings:

language="multi"

,

interim_results=False

,

endpointing=500

,

filler_words=False

,

punctuate=True

,

smart_format=True

◦ VAD:

silero.VAD.load(activation_threshold=0.35)

◦ Turn detection:

MultilingualModel()

◦ Noise cancellation:

noise_cancellation.BVCTelephony()

for SIP participants ◦

min_endpointing_delay=0.5

,

max_endpointing_delay=3.0

• Logs & metrics (attach paste) ◦ Raw STT metrics for a failed utterance (timestamps in ms and audio_duration entries). Include the full raw JSON if possible. ◦ VAD timestamps (speech detected time). ◦ The sequence of log entries showing STT metrics repeatedly with no transcript and later a successful transcript for a longer utterance (you already have examples like EventLogID 94975223). • What you’ve already tried ◦ Confirmed no krisp/NC on SIP trunk ◦ Verified subscription and handler are active ◦ Confirmed STT metrics but no

user_input_transcribed

or EOU events ◦ Willing to try disabling BVCTelephony, switching language to

"en"

, enabling

interim_results=True

, and upgrading packages • Expected vs observed behavior ◦ Expected: short replies like “Yes” produce a transcript and the bot reacts. ◦ Observed: STT metrics logged, but no transcript or EOU for the short reply; longer replies do transcribe. • Priority/impact ◦ Bot goes silent on inbound SIP short replies; impacts user experience and increases hangups. Where to send it • Preferred: Open a GitHub issue in the LiveKit agents repository and include the above content and log attachments. This allows engineers to triage, link to code, and track fixes. ◦ Repo: https://github.com/livekit/agents (paste the title, reproduction steps, logs, and configuration) • Alternate: If you prefer, include the same details in a support email or internal ticket system used by your team. Suggested issue template text (ready to paste into GitHub) • Title: STT (Deepgram nova-3) logs metrics but returns no transcripts for short inbound SIP utterances • Body: ◦ Environment: livekit-agents[silero,turn-detector]~=1.2, livekit-plugins-noise-cancellation~=0.2, LiveKit Cloud (prod) ◦ Reproduction steps: (paste steps from above) ◦ STT config:

stt=inference.STT(
      model="deepgram/nova-3",
      language="multi",
      extra_kwargs={
        "interim_results": False,
        "filler_words": False,
        "endpointing": 500,
        "punctuate": True,
        "smart_format": True,
        "numerals": True,
      },
    )
    vad=silero.VAD.load(activation_threshold=0.35)
    turn_detection=MultilingualModel()
    min_endpointing_delay=0.5
    max_endpointing_delay=3.0
    noise_cancellation=BVCTelephony() for SIP participants

• Logs: (attach or paste raw STT metrics, VAD timestamps, and EventLogIDs: 94975223, 94952280, 94955339) • Steps tried: (list what you’ve tried) • Impact: Bot remains silent and callers hang up on short replies. If you want, I can: • Draft the GitHub issue body for you using the exact logs you pasted (I’ll format it and include the EventLogIDs and STT metric table). • Point out which code lines to include or attach a small snippet to reproduce locally. Which would you like me to prepare — the full paste-ready issue body with your logs, or just the short template so you can open it yourself?

Yeah full

Title STT (Deepgram nova-3) logs metrics but returns no transcripts for short inbound SIP utterances Environment • livekit-agents[silero,turn-detector]~=1.2 • livekit-plugins-noise-cancellation~=0.2 • LiveKit hosting: LiveKit Cloud (prod) • STT provider: Deepgram (model: deepgram/nova-3) Problem summary On inbound SIP calls, short consumer replies (1–3 words such as “Yes”, “Later”, “Yeah what’s up”) trigger VAD and produce STT metrics, but no transcript or EOU is ever emitted. Longer utterances on the same calls are transcribed normally. Outbound calls with the same configuration do not exhibit this issue. The result: the bot remains silent and the caller hangs up. Reproduction steps (minimal) 1. Deploy an agent with the configuration below and register a

user_input_transcribed

handler that logs transcripts. 2. Place an inbound SIP call to the bot (LiveKit Cloud). 3. Bot plays a short greeting (e.g., “Am I speaking with Sharfuddin?”). 4. Consumer responds immediately with a short reply (1–3 words) such as “Yes” or “Later”. 5. Observe: ◦ VAD detects speech. ◦ STT metrics are logged repeatedly. ◦ No

user_input_transcribed

event and no EOU event for that short reply. ◦ Later, a longer utterance (e.g., “Hello?”) may produce a transcript. Exact configuration used • STT:

stt = inference.STT(
    model="deepgram/nova-3",
    language="multi",
    extra_kwargs={
        "interim_results": False,
        "filler_words": False,
        "endpointing": 500,
        "punctuate": True,
        "smart_format": True,
        "numerals": True,
    },
)

• VAD and turn-detection:

vad = silero.VAD.load(activation_threshold=0.35)
turn_detection = MultilingualModel()
min_endpointing_delay = 0.5
max_endpointing_delay = 3.0

• Noise cancellation: ◦ Agent-side NC via RoomOptions audio_input: ▪︎ SIP participants: noise_cancellation.BVCTelephony() ▪︎ Non-SIP participants: noise_cancellation.BVC() • SIP trunk: minimal config, no krisp/NC on trunk itself. Subscription and handlers • session.start() is used and handles subscription automatically (no manual subscription). •

user_input_transcribed

handler example (no filtering of transcripts):

@session.on("user_input_transcribed")
def _on_user_transcript(ev):
    cancel_greeting_timer("transcript received")
    cancel_dead_air_timer("transcript received")

    if not agent.amd_handler.first_transcript_received.is_set():
        transcript = ev.transcript if hasattr(ev, 'transcript') else str(ev)
        <http://logger.info|logger.info>(f"First transcript received: '{transcript}'")
        agent.amd_handler.set_first_transcript(transcript)
        agent.call_state.set("consumer_spoke", True)

Relevant logs (failed utterance example — EventLogID 94975223) • VAD detected speech at 1771024382955 (231302.955) — consumer said “Yes” shortly after bot greeting. • STT metrics (timestamps Unix ms; audio_duration shown): a. 1771024374416 (231254) — audio_duration: 2.1s — during bot greeting playout b. 1771024379249 (231259) — audio_duration: 4.8s — bot greeting finished ~231300 c. 1771024383283 (231303) — audio_duration: 4.05s — consumer said “Yes” ~231303 d. 1771024383981 (231303) — audio_duration: 0.7s — same speech region e. 1771024388982 (231308) — audio_duration: 5.0s — silence after “Yes” f. 1771024389928 (231309) — audio_duration: 0.95s — still no transcript g. 1771024394736 (231314) — audio_duration: 4.8s — consumer “Hello?” — this one DOES produce a transcript • Notes: multiple STT metric entries occur for the short “Yes” but no

user_input_transcribed

or EOU is emitted for that utterance. The first transcript appears ~11–12s later for a different (longer) utterance. Other reproducible examples (same pattern) • EventLogID 94952280 — consumer “Yeah what’s up” → 7 STT metrics, no transcript, 25s silence, hangup • EventLogID 94955339 — consumer “Later” → STT metrics, no transcript, 9s silence, hangup What I observe / diagnosis so far • STT metrics indicate Deepgram was invoked and processed audio chunks, so audio transport and subscription are functioning. • No transcript/EOU implies Deepgram returned no confident final result or only interim hypotheses that never finalized. • The multiple and sometimes long

audio_duration

values for a 1-word reply suggest VAD/audio-chunk alignment or buffering issues (VAD may be combining silence and speech or producing multiple overlapping chunks). • Using

interim_results=False

hides any partial hypotheses, so intermittent attempts by STT to transcribe short speech could be invisible. • This behavior is consistent with known limitations of STT providers on very short utterances (1–3 words), and aggressive NC (BVCTelephony) can exacerbate the problem by suppressing short or quiet speech. What I have already tried • Confirmed no krisp/NC on the SIP trunk (no double-processing). • Verified subscriptions and the

user_input_transcribed

handler are active and do not filter transcripts. • Collected raw STT metric timestamps and VAD event time. • Prepared to test disabling BVCTelephony, switching to language="en", enabling interim_results, relaxing VAD/endpointing, and/or upgrading packages. Impact • Bot remains silent for short consumer replies on inbound SIP calls, causing user frustration and hangups. This affects production LiveKit Cloud traffic. Requested help / questions for the LiveKit team 1. Can you confirm whether this behavior (STT metrics but no final transcript for short inbound SIP utterances using Deepgram nova-3) is expected given current agents + NC behavior, or whether this indicates a bug in the LiveKit pipeline (VAD/endpointing turn detection coordination, or NC integration)? 2. Are there additional STTMetrics fields we can enable/log (e.g., request_id, confidence, speech_id) from livekit-agents v1.2 or via instrumentation to correlate STT requests with Deepgram responses? If not, can the team advise how to capture these raw request/response pairs? 3. Is there guidance on optimal Deepgram parameters for short-turn detection in this environment (e.g., endpointing tuning, interim_results usage, or model-specific flags)? 4. Is there a recommended livekit-agents / noise-cancellation plugin version pair that reduces the likelihood of dropped short utterances on narrowband SIP audio? 5. Are there server-side (LiveKit Cloud) logs or traces you can inspect for the listed EventLogIDs to help identify where the finalization step is failing? Suggested next steps I can take immediately (if useful) • Disable BVCTelephony() for SIP participants and re-run the flow to confirm if NC is the cause. • Switch STT

language

from "multi" to "en" and enable

interim_results=True

to observe partial transcripts. • Upgrade livekit-agents and noise-cancellation plugin to recommended stable versions and re-test. • Add request-level logging to capture

request_id

and

duration

in STTMetrics (I can patch local logging if you suggest the best fields to capture). Attachments • EventLogIDs: 94975223, 94952280, 94955339 (include when investigating) • STT metrics snippet (from 94975223) — see the “Relevant logs” section above. I can paste the raw JSON metric entries if desired. Priority • Medium/High: impacts real caller experience and leads to dropped calls, but reproducible and isolated to short utterances. Please let me know if you want me to: • Paste the raw JSON STTMetrics entries for the EventLogIDs (I can attach them directly to the issue). • Run the quick NC-disabled test and paste the before/after metric differences. • Provide a small STT-only test agent snippet to reproduce the behavior on demand. Thank you — any guidance or actions you can take to triage whether this is a product bug vs. expected STT/provider limitation would be greatly appreciated.