I can set the stt model in the AgentSession, but how do I know that it is this model that is being used to generate the chat context?

Thanks for reaching out, Orjan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I don't receive any stt metrics even though we're actively transcriping from on_conversation_item_added. What triggers the stt metrics?

You're not receiving STT metrics — thanks for the detail that transcription is happening from

on_conversation_item_added

. Quick answers and checks: What triggers STT metrics • STTMetrics are emitted only when the STT model completes processing an audio segment and returns a final transcript (after end-of-speech/VAD). Interim/partial transcripts do not emit STTMetrics. • If you’re using a

RealtimeModel

(e.g., OpenAI Realtime), standalone STT is handled inside the model and STTMetrics are not emitted. • For streaming STT, metrics appear only on the final transcript; for non-streaming STT they appear after the batch completes. Quick checklist to diagnose 1. Confirm STT is configured on the AgentSession (not relying solely on a

RealtimeModel

). 2. Verify you're getting final transcripts (end-of-speech). If you only see INTERIM transcripts in

on_conversation_item_added

, STTMetrics won’t be emitted. 3. Ensure your metrics listener is registered and checks for STTMetrics, e.g.:

@session.on("metrics_collected")
def _on_metrics_collected(ev: MetricsCollectedEvent):
    if isinstance(ev.metrics, STTMetrics):
        print("STT Metrics received:", ev.metrics)

1. Confirm your STT provider returns final transcript events (some providers signal end-of-speech differently). If you want, tell me: • which STT model or plugin you set in

AgentSession

, and • whether you see INTERIM or FINAL transcripts in

on_conversation_item_added

and I’ll point to the exact logs/fields to inspect next. Sources: Observability data hooks | LiveKit docs | Agents (Node.js) | livekit/livekit_composite | DeepWiki

If you’re using a

RealtimeModel

(e.g., OpenAI Realtime), standalone STT is handled inside the model and STTMetrics are not emitted Tell me more about htis

You're asking for more detail about why

STTMetrics

aren't emitted when using a

RealtimeModel

(for example, OpenAI Realtime). Quick, focused explanation and options: Why STTMetrics aren't emitted with a RealtimeModel • The RealtimeModel handles STT internally as part of its multimodal audio pipeline — it takes audio input, performs transcription, and produces model outputs within the same component. • STTMetrics are only emitted for a standalone STT pipeline (an external STT plugin configured on the AgentSession). Because there is no separate STT stage to observe when the model does transcription internally, there is no STTMetrics emission. • The RealtimeModel still performs end-of-utterance/turn detection server-side and can produce transcription events (e.g., transcription-completed), but those internal events do not generate the STTMetrics object used by the AgentSession observability hooks. Options if you need STT metrics • Add a separate STT plugin to your AgentSession in addition to the RealtimeModel. This runs an external transcription pipeline that will emit

STTMetrics

(at the cost of extra processing/latency). • Rely on the RealtimeModel’s transcription-complete events and inspect the transcription content/timestamps there for whatever telemetry you need (these events won’t appear as

STTMetrics

, but they can still provide timing and transcript data). If you want, tell me: • which RealtimeModel you’re using, and • whether you prefer lower latency (single-model) or metric collection (separate STT) and I’ll recommend the exact configuration and the specific events/fields to inspect. Sources: Realtime model plugins | LiveKit Docs | Observability data hooks | LiveKit docs | LiveKit turn detector plugin | LiveKit docs | examples/voice_agents | livekit/livekit_composite | DeepWiki

we use a realtime model, which in theory should perform its own stt. But we can change the quality of the transcripts by setting the language in input_audio_transcription. How does this affect the stt of the model?

You're asking how setting

language

in

input_audio_transcription

affects the RealtimeModel's internal STT. Short answer: it guides the model’s internal transcription process and usually improves accuracy for that language; leaving it as

multi

or unspecified enables language detection but can reduce per-language fidelity. Key effects • Accuracy: Specifying a single language (e.g.,

en

) focuses the model on that language, improving recognition accuracy, punctuation, and handling of accents/noise. • Detection vs. specificity:

multi

or no language lets the model auto-detect multiple languages in-stream, useful for multilingual conversations but slightly less accurate for any one language. • End-of-utterance/turn detection: The RealtimeModel still handles turn detection server-side; language setting does not change that mechanism but can affect when transcription output is interpreted (e.g., language-specific tokenization). • No STTMetrics: These internal transcriptions do not emit

STTMetrics

. You can inspect the model’s transcription-complete events and timestamps for timing/telemetry, but they won’t appear as

STTMetrics

. When to set language • Use a specific language when audio is primarily one language and you want better accuracy and formatting. • Use

multi

when speakers switch languages frequently or you need automatic detection. If you want, tell me which model and language(s) you’re using and whether you prefer higher accuracy or automatic detection, and I’ll recommend the exact

input_audio_transcription

settings to try. Sources: Realtime model plugins | LiveKit Docs | OpenAI STT plugin guide | LiveKit docs | Speech-to-text (STT) models | LiveKit Docs