https://livekit.io logo
Join Slack
Powered by
explain the whole flow when user starts talking, t...
# ask-ai
p
explain the whole flow when user starts talking, transcription, vad, transcription delay, EoU model and the total EoU reported by livekit. complete workflow,
t
Thanks for reaching out, Adarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks for reaching out! Could you please help me understand a bit better—are you asking about the workflow for a particular LiveKit feature or maybe an integration? For instance, is it about a real-time communication scenario, a specific API, or something else? Knowing this will really help me guide you more clearly through the flow involving transcription, VAD, and EoU.
p
when a user starts speaking, how does livekit detects things, explain every component i asked you about earlier. and when he stops speaking, how they work and detect and make an llm call
t
I'm glad you asked about how LiveKit detects when a user starts and stops speaking! It uses a layered system that combines Voice Activity Detection (VAD), transcription, an End-of-Utterance (EOU) model, and LLM inference to make this process smooth and accurate. Let me walk you through the details: • Voice Activity Detection (VAD): This clever neural network listens in real time to detect speech. When it notices silence after speech, it starts a timer. If the silence lasts longer than a set threshold (usually 500ms), it recognizes that the user has stopped speaking. • Transcription: The audio is sent to a Speech-to-Text (STT) service that provides real-time transcription. You get interim transcripts as the user talks, and a final transcript once they finish. There’s a little delay here, called transcription delay, between the end of speech and the final text. • End-of-Utterance (EOU) Model: This transformer-based model looks at the meaning behind the words to guess if the user is done speaking. If it’s unsure, it cleverly extends the silence timeout so you don’t get interrupted too soon. • LLM Inference: After confirming the end of the user’s turn, an LLM call kicks in. It uses the final transcript and conversation context to generate a thoughtful response. • Turn Detection Modes: LiveKit offers several modes like VAD, STT, real-time LLM detection, and even manual control. The system smartly picks the best mode depending on what’s available. This whole workflow is designed to catch speech and silence accurately, so interruptions are minimized while keeping things responsive. If you'd like, I’m here to explain any part in more detail—just let me know! Sources: Improving voice AI's turn detection with transformers | agents/src/voice/agent_activity.ts | How does end-of-utterance detection work in conversations? | OpenAI Realtime API integration guide | LiveKit Docs
p
but in my case that transcription_delay is 400 ms
t
I completely understand how a 400ms transcription delay can feel a bit frustrating. This delay represents the time the Speech-to-Text (STT) service takes to finalize the transcription once the user stops speaking. It’s included in the overall end-of-utterance (EOU) timing and the following LLM call. If you’re noticing this delay regularly, it might be related to the specific STT service you’re using or how it’s set up. Please feel free to reach out if you'd like help optimizing this delay or if you want to chat more about how it affects your workflow. I’m here to support you!
p
yeah i would like to optimize this transcription delay
t
To help you optimize the transcription delay of 400ms, here are some friendly tips and best practices you can try: • Adjust STT Parameters: ◦ max_delay: Consider reducing the maximum delay for final transcription, maybe around 0.7 seconds, to speed things up. ◦ end_of_utterance_silence_trigger: Lowering the silence threshold before finalizing a transcript (for example, to 0.3 seconds) can also help. ◦ end_of_utterance_mode: You might want to use 
EndOfUtteranceMode.FIXED
for more predictable delays or 
ADAPTIVE
if you prefer dynamic adjustments. • Enable Partial Transcripts: Turning on 
enable_partials
allows you to receive interim transcription updates while the user is still speaking, which can make things feel much quicker. • Optimize Audio Chunk Size: Using smaller audio chunks, like 10ms frames, can improve responsiveness. For 16kHz audio, setting 
chunk_size: 160
works well. • Network Proximity: It’s a great idea to deploy LiveKit agents close to the STT service to minimize any network latency. • Choose Low-Latency STT Providers: Picking providers that are optimized for real-time transcription, such as Google Cloud STT, can make a big difference. Here’s an example configuration you might find helpful for achieving low latency:
Copy code
STT(
    max_delay=0.7,
    end_of_utterance_silence_trigger=0.3,
    end_of_utterance_mode=EndOfUtteranceMode.FIXED,
    enable_partials=True,
    chunk_size=160,
)
If you'd like, I’m here to help you apply these settings or answer any other questions you might have. Just let me know! Sources: livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py | Speechmatics STT integration guide | LiveKit Docs | Live Conversations with AI using ChatGPT and WebRTC | Agent Testing & Evaluation | livekit/livekit_composite | DeepWiki | Speech & Audio Processing | livekit/livekit_composite | DeepWiki
p
i am particularly working with deepgram STT
t
To help you get the best transcription experience with Deepgram STT and LiveKit, here are some friendly tips to optimize transcription delay: • Reduce `endpointing_ms`: Try lowering the silence duration threshold to something like 10ms or even 0ms. This helps finalize transcripts more quickly. Just a heads-up, setting it to 
0
means endpointing is disabled, so you’ll need to manually close the stream to get the final results. • Enable 
interim_results
and `no_delay`: Turning on 
interim_results=True
lets you see partial transcriptions as speech happens, and 
no_delay=True
helps avoid waiting for full utterances before showing results. • Use faster models: Choosing lightweight models like 
nova-2-general
can really help reduce latency. • Disable post-processing features: If you’re aiming for ultra-low latency, turning off 
smart_format
and 
punctuate
can make a difference. • Prefer `keyterms`: When using Nova-3 models, it’s better to use 
keyterms
instead of 
keywords
for smoother compatibility and improved recognition. Here’s a simple example configuration to get you started:
Copy code
from livekit.plugins import deepgram

stt = deepgram.STT(
    model="nova-2-general",
    interim_results=True,
    no_delay=True,
    endpointing_ms=10,  # Lower than default 25ms
    smart_format=False,  # Disable if latency is critical
    punctuate=False,
    language="en-US"
)
If you’d like, I’m here to help you apply these settings or answer any other questions you might have. Just let me know! Sources: livekit-plugins/livekit-plugins-speechmatics/livekit/plugins/speechmatics/s... | Deepgram integration guide | LiveKit Docs | Live Conversations with AI using ChatGPT and WebRTC | Live Conversations with AI using ChatGPT and WebRTC | Live Conversations with AI using ChatGPT and WebRTC
I'd love to understand a bit more about how you're using the perceived latency calculation. Are you looking to optimize it, check if it's accurate, or maybe combine it with other metrics? Also, if you're facing any particular challenges with the formula you mentioned, please feel free to share—I’m here to help!
It looks like you’ve got a solid approach to calculating perceived latency! By honing in on the timestamps that truly affect the user experience, you’re capturing the real end-to-end latency in a meaningful way. Here’s a quick rundown of why this makes sense: • Start Point: Using 
EOUMetrics.last_speaking_time
as your baseline is great because it marks the exact moment the user finishes speaking. • End Point: Calculating 
first_audio_time
with 
agent_metrics.timestamp - agent_metrics.duration + agent_metrics.ttfb
smartly represents when the first audio byte is ready to play, which aligns perfectly with when the user perceives the agent starting to respond. The formula:
Copy code
perceived_latency = first_audio_time - self.last_speaking_time
is a trustworthy way to measure the conversational latency from the user’s perspective. By focusing on these key timestamps and setting aside inner metrics, you’re avoiding confusion caused by internal processing differences. If you’d like, I’m here to help with any further questions or to validate this approach for particular scenarios—just let me know!
p
why this is better close to user percieved latency then the total latency given by livekit ? will it be conflicting with the livekit community ?
t
I really appreciate your thoughtful approach—it’s great that you’re focusing on user-perceived latency by measuring the time between when the user stops speaking (
EOUMetrics.last_speaking_time
) and when the agent’s response starts playing (
first_audio_time
). This really captures the user’s actual experience, highlighting the flow of the conversation rather than just the behind-the-scenes processing. On the other hand, the total latency reported by LiveKit includes all the internal metrics, like processing delays, which sometimes might not match what the user actually feels. For instance, if some internal metrics face delays or temporary issues, the total latency could look higher even though the user’s experience remains smooth. About any concerns with the LiveKit community, your approach actually complements their existing metrics instead of conflicting with them. It adds a valuable layer focused on the user experience, which is really important when optimizing conversational latency. Just make sure to clearly explain the purpose and scope of your perceived latency calculation, and I’m confident it will be welcomed as a user-focused enhancement. If you’d like, I’m here to help you present this to the community or to fine-tune your approach further—just let me know!
p
as i have exolained you till now, can you summarize my motive for this percieved latency ?
t
It sounds like you're aiming to measure the true end-to-end conversational latency that your users actually experience, which is fantastic! This means focusing on the time between when a user stops speaking and when the agent’s response starts playing, rather than just looking at LiveKit’s total latency metrics. Those can sometimes include internal delays that don’t really affect what the user feels. By taking this approach, you’re working towards: • Getting a clearer, more accurate picture of the latency users perceive. • Filtering out internal metrics that might make delays seem worse than they really are. • Gaining better insights into real-world interactions to help improve the overall user experience. If this sounds like what you’re going for, that’s wonderful! And if you’d like me to explain anything further or dive deeper, just let me know—I’m here to help!
p
Can you give a prompt for google gemini nano banana , to make a daigram to explane the complete EoU pipeine from users voice to partial transcript, vad and then full transcript finalized, transcription delay and then eou model triggere and then llm call. explain this thing so that i can build this system architecture diagram with ai.
t
Here’s a prompt you can use with Google Gemini Nano Banana to generate a system architecture diagram for the complete EoU pipeline:
Copy code
"Create a detailed system architecture diagram illustrating the End-of-Utterance (EOU) pipeline in a conversational AI system. The diagram should include the following components and their interactions:

1. User's voice input: Represent the audio input from the user.
2. Voice Activity Detection (VAD): Show how VAD detects the start and end of speech, triggering further processing.
3. Partial transcription: Indicate how interim transcripts are generated in real-time during speech.
4. Full transcription: Highlight the process of finalizing the transcript after the user stops speaking, including transcription delay.
5. EOU model: Depict the EOU model analyzing the transcript and predicting whether the user has finished speaking.
6. LLM call: Show how the system triggers a call to the Large Language Model (LLM) after confirming the end of the user's turn.

Include arrows to represent the flow of data and processing between these components. Use labels to indicate key processes like 'VAD triggers transcription,' 'Transcription delay,' and 'EOU model confidence score.' Ensure the diagram is clear, logical, and visually intuitive."
If you'd like, I’m here to help you refine this prompt further or add any extra details to make the diagram even clearer. Just let me know!
p
it generated user input then vad and then partial transcription and then full transcription and then so basically everything generated was serially, but as far as i understand partial transcripts are generated parallely
t
You're absolutely right—partial transcripts are generated alongside the user's speech, not one after the other following VAD. It’s important to understand that partial transcription starts as soon as the audio comes in, while VAD works on its own to detect when speech begins and ends. Here’s a friendly overview of how the flow should look: • User Input: Audio is captured and sent simultaneously to both VAD and the transcription service. • VAD: Listens for speech and signals when it starts and stops. • Partial Transcription: Happens alongside VAD, giving you interim results as the user talks. • Full Transcription: Completes the transcript once the user finishes speaking, accounting for any transcription delay. • EOU Model: Reviews the final transcript to decide if the user has finished their turn. • LLM Call: Happens after the EOU model confirms the user is done speaking. As you update your diagram, just keep in mind that partial transcription runs in parallel right from the start of the user input, while VAD and the other parts work independently but all play a role in the pipeline. If you’d like, I’m here to help you tweak the prompt or the diagram—just let me know!
p
yeah tweak the prompt
t
Here’s the updated prompt to better reflect the parallel processing of partial transcription:
Copy code
"Create a detailed system architecture diagram illustrating the End-of-Utterance (EOU) pipeline in a conversational AI system. The diagram should include the following components and their interactions:

1. User's voice input: Represent the audio input from the user.
2. Voice Activity Detection (VAD): Show how VAD detects the start and end of speech, triggering further processing.
3. Partial transcription: Indicate that interim transcripts are generated in real-time during speech, running in parallel with VAD.
4. Full transcription: Highlight the process of finalizing the transcript after the user stops speaking, including transcription delay.
5. EOU model: Depict the EOU model analyzing the transcript and predicting whether the user has finished speaking.
6. LLM call: Show how the system triggers a call to the Large Language Model (LLM) after confirming the end of the user's turn.

Ensure that partial transcription is shown as a parallel process starting immediately with user input, while VAD and other components operate independently but contribute to the overall pipeline. Use arrows to represent the flow of data and processing between these components, and label key processes like 'VAD triggers transcription,' 'Transcription delay,' and 'EOU model confidence score.' The diagram should be clear, logical, and visually intuitive."
If you'd like, I’m here to help with any tweaks or to add more details. Just let me know!
Open in Slack
PreviousNext
Powered by