I'm using a STT-LLM-TTS pipeline. For some reason, LiveKit is not starting LLM processing immediately after

on_user_turn_completed

returns. There's a 1.3-second internal delay before it begins LLM processing

Thanks for reaching out, Anas. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

here's my setup:

session: AgentSession = AgentSession(
    stt=deepgram.STT(
        model="nova-3",
        punctuate=True,
    ),
    llm=openai.LLM(
        model="gpt-4o-mini",
    ),
    tts=cartesia.TTS(),
    vad=silero.VAD.load(
        min_speech_duration=0.2,
        min_silence_duration=0.9,   
        prefix_padding_duration=0.2,
        max_buffered_speech=30.0,
        activation_threshold=0.5,
        sample_rate=16000,
    ),
    turn_detection="vad",
)

I'm glad you're reaching out about the 1.3-second delay in your STT-LLM-TTS pipeline. Let's explore some optimization steps together that might help improve the response time: • Try adjusting the

min_endpointing_delay

and

max_endpointing_delay

parameters in your

AgentSession

configuration. For instance, setting

min_endpointing_delay=0.5

can often help reduce the delay right after the user finishes speaking. • Prewarming the VAD model during initialization can be a great way to avoid loading delays. You can do this by using a

prewarm

function to load the model before the session starts, which helps things run more smoothly. • Make sure your Deepgram STT is set up for low latency by using the

nova-3

model and enabling features like

keyterms

to tailor it to your domain. • For the OpenAI LLM, using the

gpt-4o-mini

model with a

temperature=0.45

setting can provide faster and more consistent responses. • Optimizing your Cartesia TTS by selecting the

sonic-2

model and setting the speed to "fast" can speed up synthesis nicely. • Consider using the

MultilingualModel

for turn detection—it often improves how quickly the system detects when the user stops speaking. • Enabling noise cancellation through Krisp integration can really boost VAD and STT accuracy, which in turn helps reduce any reprocessing delays. Also, it might be helpful to monitor your pipeline’s latency by attaching event listeners that log metrics like Time to First Token (TTFT) and endpointing delays. If you'd like, I’m here to guide you through setting these up or to help with any specific component you want to focus on. Just let me know! Sources: agent-examples/agent_examples/pipeline_agent.py | metrics/send-metrics-to-3p/send_metrics_to_3p.py | Voice AI quickstart | LiveKit Docs | examples/voice_agents/restaurant_agent.py | Voice AI quickstart | LiveKit Docs

I'm still facing the same issue. There's a delay of 1-3 seconds before the LLM returns

I understand how frustrating delays in your pipeline can be, so let’s work through some helpful steps together to get things running more smoothly: • Measure LLM Response Time: Try using the

metrics_collected

event to track

ttft

(Time To First Token) and

tokens_per_second

. This can give us a clear picture of whether the LLM might be causing the slowdown. • Optimize STT: Make sure you’re using streaming recognition with low-latency models like Deepgram’s

nova-3

. Also, adjusting the audio chunk size to 20ms can help speed things up. • Optimize TTS: Consider using chunked streaming for TTS output, breaking it down by sentence boundaries. This often helps reduce the delay users perceive. • Network Optimization: Placing your agent close to the LLM, STT, and TTS services can really cut down on network latency. • Pipeline-Wide Improvements: Whenever possible, enabling concurrent processing so STT, LLM, and TTS can run simultaneously will improve efficiency. • User Engagement: To keep users engaged during any unavoidable delays, you might use pre-recorded audio cues or dynamic “thinking” messages. If you’d like, I’d be happy to guide you step-by-step on setting up metrics collection or focusing on optimizing any specific part of your pipeline. Just let me know how I can assist! Sources: Getting Help | livekit/livekit_composite | DeepWiki | delay-handling/README.mdoc | Capturing metrics | LiveKit Docs | tests/test_agent_session.py | Live Conversations with AI using ChatGPT and WebRTC