livekit inference and agent latency optimisation

Thanks for reaching out, Saurabh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

how to use livekit inference ?

Thanks — I’m responding to your question: “how to use LiveKit Inference?” Below is a concise, practical guide so you can get started quickly with examples, key requirements, and next steps. Quick summary • LiveKit Inference is a unified gateway to STT, LLM, and TTS models accessible with a single LiveKit API key. It’s available on LiveKit Cloud and supports Python and Node.js SDKs. • Typical flow: configure an AgentSession with the model IDs you want (STT, LLM, TTS) → deploy/run your agent → LiveKit routes inference calls to the best provider. Setup & requirements • Use supported SDK versions: ◦ Python Agents SDK v1.2.13+. ◦ Node.js Agents SDK v1.0.7+. • Create a LiveKit Cloud account and install/authenticate the LiveKit CLI (use

lk cloud auth

) to manage API keys and deployments. • Make sure your environment can reach LiveKit Cloud (or your self-hosted LiveKit if applicable). Basic usage examples • Python (LLM):

from livekit.agents import AgentSession

session = AgentSession(
    llm="openai/gpt-4.1-mini"
)

• Node.js (LLM):

import { AgentSession } from "@livekit/agents";

const session = new AgentSession({
  llm: "openai/gpt-4.1-mini"
});

• Specify STT (example: Cartesia, with language tag):

session = AgentSession(
    stt="cartesia/ink-whisper:en"
)

const session = new AgentSession({
  stt: "cartesia/ink-whisper:en"
});

Custom configuration • Use typed classes in the

inference

module to set provider or extra model parameters. Python example:

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.LLM(
        model="openai/gpt-5-mini",
        provider="openai",
        extra_kwargs={"reasoning_effort": "low"}
    )
)

Node.js example:

import { AgentSession, inference } from "@livekit/agents";

const session = new AgentSession({
  llm: new inference.LLM({
    model: "openai/gpt-5-mini",
    provider: "openai",
    modelOptions: { reasoning_effort: "low" }
  })
});

(Note: Node uses

modelOptions

instead of

extra_kwargs

.) Supported models & providers • LLMs: OpenAI, Google/Gemini, DeepSeek, Kimi (via Baseten), etc. • STT: Deepgram, AssemblyAI, Cartesia, ElevenLabs, etc. • TTS: Cartesia, ElevenLabs, etc. • For full lists and IDs, see the LLM/STT/TTS model pages in the LiveKit docs: https://docs.livekit.io/agents/models/ Key features & benefits • Unified API key and consolidated billing for multiple providers. • Global co-location and provisioned capacity to reduce latency and avoid public endpoint congestion. • Centralized concurrency management in the LiveKit Cloud dashboard. • Dynamic routing (coming soon) for automatic low-latency provider selection. Getting started resources • Voice AI quickstart: https://docs.livekit.io/agents/start/voice-ai-quickstart/ • Models overview and per-provider model pages: https://docs.livekit.io/agents/models/ • Starter repo (example): https://github.com/livekit-examples/agent-starter-python If you want a targeted setup, tell me: • Are you using LiveKit Cloud or self-hosting? • Which STT, LLM, and TTS providers or model IDs do you plan to use? • Do you want a minimal voice demo (quickstart) or a production-ready agent with monitoring and scaling? Share those details and I’ll give step-by-step instructions tailored to your environment (including CLI commands, AgentSession configs, and suggestions for latency/scale). Sources: Tom Shapland - LiveKit Blog | Models overview | LiveKit Documentation | Voice AI quickstart | LiveKit Documentation | OpenAI and LiveKit | LiveKit Documentation | Cartesia STT | LiveKit Documentation

most used config for inference for production livekit agents?

Thanks — I’m responding to your question about the “most used config for inference for production LiveKit agents.” Below are practical, production‑grade configuration recommendations that teams commonly use to balance latency, reliability, and cost. I’ll list defaults you can start with and optional adjustments for higher scale or lower latency. Core principles (what most production setups optimize for) • Low, consistent latency (network co‑location + provisioned inference capacity). • Predictable compute (non‑burstable instances / warm serverless). • Fast turn detection without false triggers. • Observability for TTFT / TTFB and provider bottlenecks. • Resource isolation and prewarmed models to avoid cold starts. Recommended production configuration (default starting point) • Platform ◦ Deploy on LiveKit Cloud and enable LiveKit Inference to get private backbone routing and provisioned capacity. ◦ If self‑hosting, co‑locate agents in the same region as your chosen model providers. • SDK versions ◦ Python Agents SDK v1.2.13+ or Node.js Agents SDK v1.0.7+. • Agent runtime ◦ Use non‑burstable instances (e.g., m5/c5/c6i families) or LiveKit serverless with cold‑start prevention enabled. ◦ Run each agent in a dedicated container to avoid contention. ◦ Pre-download model assets where applicable and run agents as long‑running processes (not short‑lived startups). • Inference selection (typical model choices) ◦ LLM: start with a low‑latency model (example IDs you might use:

openai/gpt-4.1-mini

or a real‑time optimized family). Tune up to larger models only when you need capability over latency. ◦ STT: low‑latency, streaming STT like

cartesia/ink-whisper:en

or Deepgram for real‑time fidelity. ◦ TTS: provider/model that supports streaming audio and small chunk sizes (Cartesia/ElevenLabs variants). • AgentSession example (sensible production defaults) ◦ Use explicit inference classes when you want provider or parameter control: ▪︎ Python:

from livekit.agents import AgentSession, inference

      session = AgentSession(
        stt=inference.STT(model="cartesia/ink-whisper:en"),
        llm=inference.LLM(model="openai/gpt-4.1-mini"),
        tts=inference.TTS(model="cartesia/voice-v2")
      )

• Node.js: same mappings, using

modelOptions

for extra parameters. • Turn detection & VAD ◦ Prewarm VAD and STT processes on startup. ◦ Typical turn detection tuning: ▪︎ `min_endpointing_delay`: low value to reduce wait after speech end (example starting: 50–150 ms). ▪︎ `false_interruption_timeout`: short but forgiving (example starting: 150–300 ms) to avoid prematurely cutting users off. ◦ Enable preemptive generation when safe for your use case to start LLM inference before the user fully finishes speaking. • Concurrency & scaling ◦ Use LiveKit Cloud concurrency controls (provisioned capacity) to guarantee model throughput. ◦ Horizontally scale agents by spinning additional containers/instances rather than overloading a single large host. • Instance and container sizing ◦ CPU/memory tuned to model and STT/TTS workload; prefer steady VM families (m5/c5) sized to handle peak concurrency per container. ◦ Avoid burstable `t3`/`t4g` families due to CPU‑credit throttling risk. • Networking ◦ Keep agents and model providers in same region. If using LiveKit Cloud, leverage its co‑location/prioritized routing. ◦ Use secure, high‑throughput NICs and monitor network queueing. • Monitoring & alerts ◦ Instrument and alert on: ▪︎ Time To First Token (TTFT) — from end of user speech to first LLM token. Target < 300 ms for low‑latency experiences. ▪︎ Time To First Byte (TTFB) for TTS audio. Target < 200 ms where possible. ▪︎ Request error rates, provider latency percentiles (p50/p95/p99). ▪︎ CPU, memory, and container start times (to detect cold starts). ◦ Use Agents Metrics API and LiveKit observability dashboards to map stage‑by‑stage latency (STT → LLM → TTS). • Failover and provider diversification ◦ Configure fallback providers/models for each inference type (STT/LLM/TTS) so you can reroute if a provider has high latency. ◦ Use LiveKit Inference’s provider selection to prefer low‑latency routes. • Billing & cost controls ◦ Track per‑model consumption via LiveKit Cloud billing/usage dashboards. ◦ Use smaller, faster models for real‑time interactions; reserve larger models for background or high‑value tasks. Optional tuning for lower latency (aggressive) • Turn detection: lower

min_endpointing_delay

further and favor preemptive generation. • Use ultra‑low latency STT providers (streaming STT) and LLMs optimized for real‑time. • Pin agents in very low network‑hop regions (same AZ as model provider). • Increase provisioned concurrency and use dedicated instances for hot paths. Optional tuning for cost efficiency (conservative) • Use smaller LLMs for baseline conversation and selectively call larger models for complex tasks (routing by intent). • Batch background tasks and non‑real‑time processing to cheaper compute. • Scale down reserved capacity during off‑peak hours. Checklist to validate production readiness • Agents warm on deploy (no frequent cold starts). • TTFT and TTFB steady and within targets at p95. • No CPU throttling (no burstable instances unless monitored). • Fallback providers configured and tested. • Monitoring/alerts cover latency, errors, and cold starts. If you’d like, I can: • Generate a copy‑paste AgentSession config tuned to your expected concurrency and latency targets. • Recommend instance/container sizing given expected calls per second. • Walk through a staging checklist (deploy → smoke tests → latency profiling). Tell me which of the following applies and I’ll produce a tailored config: • You’re on LiveKit Cloud or self‑hosting? • Expected concurrent calls per second and average call duration. • Primary STT/LLM/TTS providers you plan to use.

Best balanced STT of deepgram, and tts of eleven labs

Thanks — I’m responding to your request for a “best balanced” production setup using Deepgram for STT and ElevenLabs for TTS. Below are practical, production‑ready recommendations you can apply immediately, plus sample AgentSession configs and monitoring/tuning guidance. Why this pairing • Deepgram: low‑latency, streaming STT well suited for real‑time voice agents. • ElevenLabs: high‑quality streaming TTS with natural voices and reasonable generation latency. This combination balances responsiveness and audio quality for conversational agents. Recommended production configuration (balanced for latency, cost, and quality) • Platform ◦ Use LiveKit Cloud with LiveKit Inference enabled so STT/LLM/TTS calls route over the private backbone and get provisioned capacity. • SDKs ◦ Python Agents SDK v1.2.13+ or Node.js Agents SDK v1.0.7+. • Models (start here) ◦ STT: Deepgram streaming model (use the low‑latency streaming variant your LiveKit Inference catalog exposes). ◦ TTS: ElevenLabs streaming voice model (choose a natural voice, e.g., a mid‑complexity voice rather than the largest, slowest premium voices). • AgentSession examples ◦ Python:

from livekit.agents import AgentSession, inference

    session = AgentSession(
      stt=inference.STT(model="deepgram/streaming"),   # replace with exact model id from LiveKit docs
      llm=inference.LLM(model="openai/gpt-4.1-mini"), # or your preferred low-latency LLM
      tts=inference.TTS(model="elevenlabs/voice-v2")  # replace with chosen voice id
    )

• Node.js:

import { AgentSession, inference } from "@livekit/agents";

    const session = new AgentSession({
      stt: new inference.STT({ model: "deepgram/streaming" }),
      llm: new inference.LLM({ model: "openai/gpt-4.1-mini" }),
      tts: new inference.TTS({ model: "elevenlabs/voice-v2" })
    });

• Replace model IDs with the exact IDs listed in LiveKit’s model pages for Deepgram and ElevenLabs. Turn detection & VAD • Prewarm VAD and STT processes on startup to avoid cold starts. • Suggested starting tuning: ◦ min_endpointing_delay: 50–150 ms (lower gives faster responses; increase if you see fragmenting). ◦ false_interruption_timeout: 150–300 ms (avoids accidental cutoffs). • Enable preemptive generation where appropriate to start LLM work while the user is finishing their utterance. Container/instance choices • Use non‑burstable instances (m5/c5/c6i) or LiveKit serverless with cold‑start prevention. • Run each agent in a dedicated container with predownloaded assets; keep agents long‑running rather than ephemeral. Concurrency & scaling • Horizontally scale by spinning additional agent containers instead of overloading a single host. • Use LiveKit Cloud’s provisioned concurrency to guarantee throughput to Deepgram/ElevenLabs. • Set per-agent max concurrency based on CPU and network profiling; typical starting point is 5–20 concurrent calls per medium instance, then tune. Networking & region • Co‑locate agents with your provider region (same region/AZ) or use LiveKit Cloud co‑location. • Ensure low‑latency NICs and monitor packet loss — network issues amplify STT/TTS latency. Monitoring & metrics (must-haves) • Track Time To First Token (TTFT) — end of user speech to first LLM token. • Track Time To First Byte (TTFB) for TTS audio. • Monitor STT latency (streaming lag), LLM latency, and TTS generation time separately to pinpoint bottlenecks. • Alert on p95/p99 latency increases and on error rates for Deepgram/ElevenLabs calls. Quality vs latency tradeoffs • If you need lower latency: choose lighter ElevenLabs voices and lower TTS chunk sizes, and prefer lower‑latency STT model variants. • If you need higher quality: pick richer ElevenLabs voices and allow slightly higher buffering (but measure TTFB impact). Failure handling & fallbacks • Configure a fallback TTS voice/model (another ElevenLabs voice or a lighter Cartesia voice) if response time exceeds a threshold. • Retry logic: exponential backoff for transient provider failures; consider failing over to alternate provider if latency > threshold. Operational checklist before production • Validate TTFT and TTFB at expected load (p50/p95/p99). • Confirm no CPU throttling (avoid burstable instances). • Verify agents stay warm and do not cold-start under normal traffic. • Test fallback path and error handling for both STT and TTS. If you want, I can: • Provide exact model IDs from the LiveKit model pages for Deepgram and ElevenLabs. • Generate a ready-to-copy AgentSession config tuned for a target concurrency (tell me expected concurrent calls and region). • Suggest instance/container sizing based on expected calls per second and average call duration. Which of these would be most helpful next?