https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# ask-ai
s
This message was deleted.
t
Thanks for reaching out, Maxence. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I can definitely help optimize your agent for performance and lower latency. I don't see the code attached yet, so I can't give specific recommendations. To get the most useful, targeted advice, could you please share: • The agent's code (or the relevant parts) — startup, audio/video capture/encode, network/send/receive, and any main event loop or threads. • The environment: are you running in a browser, Node, mobile (iOS/Android), or a server? • Which SDK or libraries you're using (for example LiveKit Web SDK, native SDKs, or custom WebRTC stack). • The symptoms you see: high CPU, audio/video stuttering, frame drops, packet loss, or long end-to-end latency (and approximate numbers). • Network conditions that reproduce the issue (LAN, Wi‑Fi, mobile data) and whether you see differences across networks. • Typical session size and topology (1:1, group room with N participants, SFU vs. MCU). • Any existing measurements or logs (bitrate, CPU profile, browser perf traces, WebRTC stats). • If applicable, your desired tradeoffs — e.g., reduce latency even if CPU increases, or reduce CPU and bandwidth while keeping acceptable quality. If you prefer, paste just the core pieces first (capture/encode/send loop and any throttling or reconnection code). Once I see the code and environment, I’ll: 1. Point out immediate, actionable code changes to reduce latency and CPU. 2. Recommend tuning parameters (bitrate, frame rate, resolution, keyframe interval, encoder settings). 3. Suggest architectural changes (adaptive bitrate, simulcast, transport settings, thread/worker use). 4. Provide debugging steps and the exact metrics to collect (WebRTC getStats fields, CPU sampling, pcap/simulcast stats) to validate improvements. Send the code and the environment details and I’ll get started.
Thanks — this is a great code sample to work from. I’m responding to your request for ways to improve performance and lower latency in the STT → LLM → TTS voice-agent pipeline you pasted. Below are focused, actionable changes and checks you can make right away, grouped by impact and how to apply them to your code. High-impact changes (start here) • Use LiveKit Inference instead of direct ElevenLabs/OpenAI plugin calls. ◦ Why: co-locates model execution, reduces public internet hops, and uses provisioned capacity for lower and more consistent latency. ◦ How to apply: replace the 
elevenlabs.STT
/ 
elevenlabs.TTS
/ direct OpenAI LLM instantiation with inference wrappers (example below). Keep your existing session wiring; just swap the instances given to 
AgentSession
. ▪︎ Example pattern:
Copy code
from livekit.agents import inference

      stt_instance = inference.STT(model="elevenlabs/stt")
      tts_instance = inference.TTS(model="elevenlabs/tts", voice=lang_config["tts_voice_id"])
      llm_instance = inference.LLM(model=OPENAI_LLM_MODEL)
• Note: ensure your environment and LiveKit config allow inference routing and that API keys/region settings are correct. • Deploy your agent close to the AI services (same cloud region). ◦ Why: network round-trip is commonly the largest contributor to pipeline latency. ◦ Action: run your agent in the same region as your STT/LLM/TTS endpoints or use LiveKit Inference hosted near those models. Medium-impact tuning (quick to change) • Keep and validate VAD pre-warm (you already do this). ◦ Confirm 
server.setup_fnc = prewarm
runs for every worker process and that 
proc.userdata["vad"]
is available quickly at session start. • Keep 
preemptive_generation=True
(you already set this). ◦ This is correct: it reduces perceived latency by starting model generation earlier. • Turn-detection timing: tune 
min_endpointing_delay
. ◦ Why: lowers wait time for end-of-turn detection while avoiding premature truncation. ◦ Try lowering from default to 0.2s and measure:
Copy code
session = AgentSession(..., min_endpointing_delay=0.2)
• Avoid heavy noise cancellation on CPU-constrained hosts. ◦ You currently select 
BVC()
vs 
BVCTelephony()
based on participant kind. If CPU is a bottleneck, test disabling or choosing a less expensive NC option for the agent process. Lower-impact/operational improvements • Use non-burstable CPU instances for production (avoid t3/t4g). ◦ Sustained CPU guarantees reduce throttling during inference or audio processing bursts. • Keep your fallback minimal. ◦ Your 
_say_error_and_disconnect
is good; ensure it only instantiates TTS/LLM for short messages so it doesn’t add load. Observability: measure before/after • Add precise metrics to validate improvements (these are the ones to collect and compare): ◦ STT latency (audio chunk → transcript) ◦ LLM TTFT (time to first token) ◦ TTS TTFB (time to audio chunk) ◦ End-to-end response time (user stop speaking → audio playback start) ◦ CPU and memory per process ◦ VAD latency / false endpoints • You already call `setup_metrics(...)`; ensure you capture the above fields and graph them per deployment region/instance type. Code-level micro-optimizations to check in your implementation • Minimize per-request allocations and object creation in the hot path: ◦ Reuse long-lived model/HTTP clients (you already create these once per session — good). ◦ Ensure nothing in your audio pipeline re-initializes on every chunk. • Batch small audio frames when sending to STT (if supported) to reduce request overhead while keeping latency acceptable. • If using HTTP-based STT/LLM/TTS APIs, enable/verify keep-alive connections and large enough connection pools. Concurrency & worker model • Make sure your worker processes are sized so that each process has enough CPU to handle one active session without contention. • If you pre-warm model artifacts, ensure each worker process has its own warmed instance to avoid cross-process contention. Quick checklist to run now • [ ] Swap direct plugin objects for LiveKit Inference objects (test on staging). • [ ] Deploy agent beside the model endpoints / inference service region. • [ ] Lower 
min_endpointing_delay
to ~0.2s and validate turn correctness. • [ ] Profile CPU while running a session; disable heavy NC if CPU saturates. • [ ] Ensure HTTP clients use keep-alive and connection pooling. • [ ] Capture the TTFT/TTFB and end-to-end timings in 
setup_metrics
. If you want, I can: • Review a short trace of WebRTC/agent metrics (TTFT, TTFB, end-to-end) and point to the largest bottleneck. • Help convert your current 
elevenlabs
/ 
openai.responses.LLM
instantiation to the exact 
inference
classes for your environment (I can produce a small diff/patch). • Recommend specific instance sizes or autoscaling rules if you share expected concurrency and throughput. Which of the above would you like to try first — switching to LiveKit Inference or capturing a short set of metrics so we can identify the primary bottleneck? Sources: How can I reduce latency in voice agents using STT, TTS and LLM? | LiveKit | Live Conversations with AI using ChatGPT and WebRTC | Pipeline Translator Agent | LiveKit Documentation | VAD Metrics | LiveKit Documentation | Introducing LiveKit Inference: A unified model interface for voice AI
Thanks — this trace is very helpful. I’ll walk through what I see, call out the main bottlenecks, and give targeted next steps you can take now. What I’m responding to • Your trace showing startup, STT partial transcripts, EOU detection, preemptive generation, LLM/TTS timings, and an early MCP error. Key findings (short) • Cold start: worker → process initialized took ~4.48s (083723.9 → 083728.4). This adds to first-response latency. • MCP failure: 
failed to list tools from MCP server
(083728.636) due to a CancelledError against http://localhost:8001/mcp — tools didn’t load. • STT is the dominant latency: transcript_delay = 2.253s (083749.060) — this is the largest contributor to end-to-end delay. • LLM & TTS are good once warmed: ◦ LLM for the user query: TTFT = 0.407s, duration = 1.316s (083750.379) — solid for real-time. ◦ TTS: TTFB ~0.49s, audio_duration ~3.53s (083750.899) — also good. • End-to-end (user stop → audio start) ≈ 3.6s — dominated by the STT delay and initial cold-start effects. Actionable next steps (priority order) 1. Fix MCP server URL / connectivity (high) ◦ The agent is trying to reach 
<http://localhost:8001/mcp>
, which fails here. Update your MCP config to the correct reachable URL for your environment so tools load and any tool-related latency/fallback logic won’t interfere. 1. Reduce cold-starts (high) ◦ Run in production / pre-warmed mode rather than console/dev for latency-sensitive flows so worker/process initialization isn’t part of the user-visible path. 1. Reduce STT latency (high — largest win) ◦ Co-locate the agent with your STT provider (same cloud region or LiveKit Inference). ◦ Use the STT provider’s low-latency real-time endpoint and ensure you use keep-alive/pooled HTTP connections. ◦ If possible, batch extremely small audio frames only slightly (tradeoff tests) or use streaming STT to reduce request overhead. 1. Keep preemptive generation and EOU model (already enabled — keep) ◦ You have 
preemptive_generation
and transformer-based EOU detection in use; these are correctly reducing perceived latency. You can keep them and tune 
min_endpointing_delay
. 1. Tune endpointing and audio processing (medium) ◦ Try lowering 
min_endpointing_delay
to ~0.2s (from 0.5) and validate for false truncation. ◦ If CPU is constrained, disable or reduce heavy noise-cancellation on the agent host. 1. Avoid burstable CPU types in production (medium) ◦ Use instances with sustained CPU for consistent inference/audio processing. 1. Add/verify metrics to narrow future investigations (medium) ◦ Capture STT latency, LLM TTFT, TTS TTFB, end-to-end time, and per-process CPU during tests. Compare pre/post changes. Quick checklist you can run now • [ ] Update MCP URL and restart agent. • [ ] Run a warm deployment (pre-warmed workers) and measure first-response. • [ ] Deploy agent in same region as STT (or use LiveKit Inference). • [ ] Capture the STT trace and a TTFT/TTFB sample after making the above changes. If you want, I can: • Provide an exact config snippet for changing the MCP URL and verifying connectivity. • Help convert your STT/TTS/LLM instantiation to LiveKit Inference objects (small diff). • Review a 30–60s metric trace (STT latency, TTFT, TTFB, CPU) and point to the single biggest remaining bottleneck. Which help would you like first — a config snippet to fix MCP, or the LiveKit Inference conversion? Sources: Server startup modes | LiveKit Documentation | How to build an agent with speech-to-text input and text-only output | Live... | Improving voice AI&amp;#39;s turn detection with transformers | How does end-of-utterance detection work in conversations? | Agent speech and audio | LiveKit Documentation
b
How is transcript_delay computed
t
Good question — thanks for sharing the trace. I’ll explain exactly what transcript_delay means and how it’s computed so you can interpret your logs and target the right bottlenecks. What transcript_delay represents • transcript_delay is the measured time between when the agent detects the end of the user’s utterance (EOU) and when the final transcript arrives from the STT service. In short: it captures the STT-side latency after speech has ended. How it’s computed (step-by-step) 1. The agent’s end-of-utterance (EOU) detector signals that the user stopped speaking. The time of that event is recorded (EOU timestamp). 2. The agent waits for the STT service to return the final transcription for that utterance. The time when the final transcript is received is recorded (transcript timestamp). 3. transcript_delay = transcript timestamp − EOU timestamp What that number includes • The time the STT provider takes to finalize and return the transcript after the utterance ended (network round trips, STT processing). • Any small local buffering/processing between the EOU event and the moment the agent records the transcript arrival. What it does NOT include • The time spent while the user is speaking (capture/streaming before EOU). • LLM/TTS times (those are separate metrics like TTFT and TTFB). Why it matters • A high transcript_delay means the STT pipeline (or its network path) is the dominant contributor to perceived latency after speech ends. This is consistent with your trace where transcript_delay ≈ 2.25s and dominates your ~3.6s end-to-end time. How to use it to debug • If transcript_delay is high, check: ◦ STT provider: use a low-latency/real-time endpoint or change region. ◦ Network: co-locate agent with STT provider or use LiveKit Inference. ◦ Streaming config: ensure small frames/streaming mode is used rather than many small HTTP requests. • Correlate transcript_delay with EOU detection time (end_of_utterance_delay) to separate detection vs STT processing issues. If you’d like, I can: • Show how to log both the EOU timestamp and transcript arrival timestamp from your agent code so you can confirm the components contributing to transcript_delay. • Review a snippet of your STT streaming code or provider config and suggest immediate tuning changes. Which would help you next — a short code snippet to add EOU vs transcript timestamps to your logs, or a review of your STT provider/endpoint settings? Sources: STT Metrics | LiveKit Documentation | Speechmatics STT plugin guide | LiveKit Documentation
b
Show how to log both the EOU timestamp and transcript arrival timestamp from your agent code so you can confirm the components contributing to transcript_delay.
t
Thanks — below is a concise, ready-to-use example showing how to log both the end-of-utterance (EOU) timestamp and the final transcript arrival timestamp inside a LiveKit agent. This will let you compute 
transcript_delay
locally and confirm which component (EOU detection vs STT finalization) is contributing to latency. What this does • Logs when EOU is detected (EOU timestamp). • Logs when the final transcript arrives (transcript arrival timestamp). • Computes transcript_delay = transcript_arrival_timestamp − eou_timestamp and prints it. Example (Python)
Copy code
import time
from datetime import datetime
from livekit.agents import Agent, AgentSession
from livekit.agents.metrics import EOUMetrics
# replace with your STT plugin import, e.g. deepgram, elevenlabs, etc.
from livekit.plugins import deepgram

class TimingAgent(Agent):
    async def on_enter(self):
        # last_eou stores the epoch seconds when EOU was detected
        self.last_eou_ts = None

        # Handler for EOU metrics emitted by the STT pipeline
        def handle_eou(metrics: EOUMetrics):
            # metrics.timestamp is seconds since epoch
            self.last_eou_ts = metrics.timestamp
            human = datetime.fromtimestamp(metrics.timestamp).strftime("%H:%M:%S.%f")[:-3]
            print(f"[EOU] detected at {human} (end_of_utterance_delay={metrics.end_of_utterance_delay:.3f}s)")

        # Subscribe to EOU metrics from the session's STT
        # Note: some STT plugin/event setups may differ; this follows the common pattern.
        self.session.stt.on("eou_metrics_collected", handle_eou)

        # Handler for final transcript arrival
        @self.session.on("user_input_transcribed")
        def handle_transcript(transcript_event):
            # transcript_event.timestamp may not exist across all plugin shapes;
            # fall back to current time if necessary
            arrival_ts = getattr(transcript_event, "timestamp", None) or time.time()
            human_arrival = datetime.fromtimestamp(arrival_ts).strftime("%H:%M:%S.%f")[:-3]

            # only compute on final transcripts
            if getattr(transcript_event, "is_final", False):
                if self.last_eou_ts:
                    delay = arrival_ts - self.last_eou_ts
                    print(f"[TRANSCRIPT] final arrived at {human_arrival} | transcript_delay={delay:.3f}s | text='{transcript_event.transcript}'")
                else:
                    print(f"[TRANSCRIPT] final arrived at {human_arrival} | (no prior EOU) | text='{transcript_event.transcript}'")

# Example session setup (minimal)
async def run_session():
    session = AgentSession(
        stt=deepgram.STT(),  # swap for your configured STT plugin
    )
    agent = TimingAgent()
    await session.start(agent=agent)
    # keep running / connect to room as you normally would

# If you use an AgentServer/entrypoint, install similar handlers inside your Agent's on_enter
Notes and troubleshooting • Some STT plugins may provide explicit 
timestamp
fields on the transcription event; others may not. In the example above I fall back to 
time.time()
to ensure you still get a reliable arrival time. • Make sure your STT plugin emits the 
eou_metrics_collected
event. If your plugin uses a different event name, subscribe to that instead. • Time sources: both timestamps use the host system clock. If your agent runs across multiple machines/services, ensure clocks are synchronized (NTP) for accurate comparisons. • Use these logs to separate: ◦ end_of_utterance_delay (time from last audio to EOU detection) ◦ transcription_delay (time from EOU → final transcript arrival) ◦ transcript_delay is the latter; compare both to identify whether EOU detection or STT finalization is dominant. If you want, I can: • Create a small patch tailored to your STT plugin (paste which plugin you use: deepgram, elevenlabs, assemblyai, etc.). • Show how to emit these measurements into your existing metrics pipeline (prometheus/grafana) instead of only printing to logs. Sources: STT Metrics | LiveKit Documentation
b
"Make sure your STT plugin emits the 
eou_metrics_collected
event. If your plugin uses a different event name, subscribe to that instead." what is the vent for elevenlabs plugin?
t
Good question — the ElevenLabs STT plugin uses the same standard LiveKit Agents event for end-of-utterance metrics: 
eou_metrics_collected
. You can subscribe to it the same way as with other STT plugins. Example (JavaScript/TypeScript):
Copy code
// subscribe to EOU metrics from the session's STT
session.stt.on("eou_metrics_collected", (metrics) => {
  console.log("EOU Metrics:", metrics);
  // metrics.timestamp (epoch seconds), metrics.end_of_utterance_delay, etc.
});
Or the equivalent pattern in Python inside an Agent:
Copy code
def handle_eou(metrics):
    print(f"[EOU] ts={metrics.timestamp}, end_of_utterance_delay={metrics.end_of_utterance_delay}")

session.stt.on("eou_metrics_collected", handle_eou)
Notes • The event includes fields like 
timestamp
(EOU detection time) and 
end_of_utterance_delay
. Use 
timestamp
as the EOU timestamp when computing transcript_delay. • If your transcription event does not provide a 
timestamp
, record the arrival with 
time.time()
(Python) or 
Date.now()/Date.now()/performance.now()
as a fallback. • If you want, paste the snippet of how you currently subscribe to transcription events and I’ll provide a small patch that logs both EOU and final-transcript arrival together. Sources: STT Metrics | LiveKit Documentation
3 Views
Open in Slack
PreviousNext
Powered by