Comparing self-hosted to LiveKit Cloud When building with LiveKit, you can either self-host the open-source server or use the managed LiveKit Cloud service: Self-hostedLiveKit Cloud*Realtime media (audio, video, data)*Full supportFull support*Egress (recording, streaming)*Full supportFull support*Ingress (RTMP, WHIP, SRT ingest)*Full supportFull support*SIP & telephony*Full supportFull support including native telephony support for fully managed LiveKit Phone Numbers*Agents framework*Full supportFull support, including managed agent hosting.*Agent Builder*N/AIncluded*Built-in inference*N/AIncluded*Who manages it*YouLiveKit*Architecture*Single-home SFUGlobal mesh SFU*Connection model*Single server per roomEach user connects to the nearest edge.*Max users per room*Up to ~3,000No limit*Analytics & telemetry*Custom / external.LiveKit Cloud dashboard*Uptime guarantees*N/A99.99% explain what is agent builder , Architecture, and Connection model with proper example

Thanks for reaching out, Developer. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

explain connection model in terms of : if i wish to parallel calls for one agent ( one use -case ) vs mutiple agents running ( multiple use-cases )

Thanks — I’ll explain the connection model specifically for these two scenarios: (A) parallel calls to a single agent (one use-case) and (B) many different agents running at once (multiple use-cases). I’ll show how jobs are dispatched, how scaling works, and give short examples so you can map this to your deployment choices. Overview (short) • The Agents framework runs agent logic inside one or more agent server processes that register with LiveKit and accept “jobs” (one job = one call/session). • Dispatching is by job: LiveKit routes a room/session job to any agent server that is eligible to accept it. • Eligibility can be general (any available agent server) or targeted by

agent_name

so only servers for a specific agent type accept that job. A) Parallel calls to one agent (single use-case) • How it connects: ◦ You deploy one agent type (single

agent_name

). Multiple agent server instances register for that

agent_name

. ◦ Each incoming room/session becomes a separate job. LiveKit dispatches each job to any available agent server registered for that agent. • Concurrency model: ◦ A single agent server instance can run many jobs in parallel (each job isolated via JobContext/process). ◦ Load is balanced across available agent server instances (round-robin / distribution). • Scaling & resilience: ◦ Scale horizontally by adding more agent server instances for that agent. Jobs are automatically distributed. ◦ If one agent server fails, pending jobs will be retried or dispatched to other servers. • Example: ◦ You have a “customer-support-bot” agent. 200 customers call simultaneously. LiveKit creates 200 jobs. If you run 4 agent server instances and each handles 50 concurrent jobs, calls are spread across them. To handle 400 concurrent calls you add more instances. B) Multiple agents running simultaneously (multiple use-cases) • How it connects: ◦ Each agent type has a unique identifier (e.g.,

survey-agent

,

drive-thru-agent

,

triage-agent

). ◦ Jobs can be dispatched explicitly for a specific

agent_name

(API, SIP rules, or token-based configuration). ◦ Only agent servers registered with that

agent_name

will receive jobs for that agent. • Concurrency & isolation: ◦ Each agent type maintains its own pool of agent servers and runs jobs independently. ◦ Resource, model choices (STT/LLM/TTS), and runtime can differ per agent type, so one agent’s heavy workload won’t directly starve another. • Scaling & operational flexibility: ◦ You scale each agent pool independently based on traffic patterns and SLAs. ◦ You can deploy different versions/configurations for each agent without affecting others. • Example: ◦ During lunch,

drive-thru-agent

gets a surge while

medical-triage-agent

remains light. You scale the drive-thru agent server pool up for that period without touching the triage servers. A room configured for

drive-thru-agent

will only dispatch to servers in that pool. Comparison summary (quick) • Single agent, many parallel calls: many jobs routed to the same agent pool → scale by adding more instances of that agent server. • Multiple agents, many use-cases: jobs routed to targeted agent pools by

agent_name

→ scale and configure each agent pool independently. Practical notes to guide decisions • If one agent type will handle the majority of traffic, size and autoscale that agent pool accordingly. • If different use-cases have different model or latency needs, separate them into different agent_names so you can tune resources independently. • Use explicit dispatch (API or rules) when you need precise control over which agent runs a session; use broader registration when any agent instance can handle a job. Would you like a recommended autoscaling strategy (metrics to use, example instance counts) for either: 1) a high-concurrency single-agent deployment, or 2) a multi-agent setup with uneven traffic across agents? Tell me which scenario (1 or 2) and approximate peak concurrent sessions and I’ll suggest concrete scaling guidance. Sources: Self-hosted deployments | LiveKit Documentation | Agent dispatch | LiveKit Documentation | Agent server overview | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki | Introduction | LiveKit Documentation

if I'm running self hosted livekit server on my "CPU VM". Dynamic class of "one agent" is running on self hosted livekit server running docker compose, which is connected with my backend. So, prompt is fetched from backend, and then that agent class starts running. For now, I have not tested this dynamic class. But, i wish to understand without using "dynamic dispatch agent name"If this dynamic agent class works. then with one livekit server running and one dynamic class agent running on CPU VM. How many parallel calls ( session ) concurrently I would be able to make?

Thanks — I understand you want a practical answer: if you run a self‑hosted LiveKit server on a single CPU VM and a single dynamic agent class (no agent_name dispatching) handles every incoming call, how many concurrent sessions can that setup support? Short answer • You should expect roughly 10–25 concurrent sessions on a single 4‑core / 8 GB VM as a reasonable starting estimate. Actual throughput may be higher or lower depending on your VM specs, agent logic, and media/model work each session requires. Why that range? Key factors that determine concurrency • VM resources: CPU cores and RAM are the biggest limits. More cores and memory → more parallel jobs. A 4‑core/8GB VM is commonly suggested as a baseline and has demonstrated ~10–25 concurrent jobs in practice. • Per‑session CPU work: STT, LLM inference calls, TTS, audio encoding/decoding, noise suppression, and any synchronous API calls all increase CPU and latency per session. • Session duration and state: Agents are stateful; long sessions hold resources longer, reducing throughput. • I/O and network: If your agent frequently calls external APIs or models, network latency and bandwidth will affect how many sessions can be active without piling up CPU wait times. • LiveKit server load behavior: The agent server runs jobs in isolated JobContexts and can be configured to stop accepting new jobs when load thresholds are hit (drain behavior and custom

load_fnc

available for self‑hosted). • Media configuration: Higher audio sampling rates, additional tracks, or video will raise resource use; turning off unneeded features reduces load. Concrete evidence and examples • A recommended starting config is 4 cores + 8 GB RAM; that often supports 10–25 concurrent voice jobs depending on what each job does. • A load test example showed ~30 agents on a 4‑core/8GB machine peaked at ~3.8 cores and ~2.8GB RAM, indicating CPU is typically the bottleneck as concurrent jobs increase. What I recommend you do next (practical steps) 1. Share your VM specs (cores, RAM), typical session profile (average call length, use of STT/LLM/TTS, synchronous API calls), and expected peak concurrent sessions — I can give a tighter estimate. 2. Run a focused load test with your actual agent code: a. Simulate realistic sessions (media + any external API/model calls). b. Monitor CPU, memory, load average, and response latency. c. Increase concurrent sessions until CPU or latency becomes unacceptable. 1. Tune: reduce media quality or disable nonessential processing, add model batching or async calls, and adjust agent server `load_fnc`/`load_threshold` to stop accepting new jobs when load is high. 2. Scale horizontally: if one VM hits limits, add more agent server instances and let LiveKit distribute jobs across them. If you want, tell me: • Your VM CPU/RAM, • Whether your agent does STT/LLM/TTS locally or calls external services, • Expected average session length and peak concurrent sessions you plan for — and I’ll give a more specific concurrency estimate and a sample load test plan you can run to validate capacity. Sources: Self-hosted deployments | LiveKit Documentation | An open source stack for real-time multimodal AI | Server options | LiveKit Documentation | Going beyond a single-core | Optimizing Video Calls for Low-Powered Devices in LiveKit (JS SDK ≥ v2.14.0...