my question is to how to make my self hosted livekit horizionatlly scaleable i have one server on aws t2.large hosting livekit + shared reddis to store voice ai calls configurations and one network loadbalncer attached livekit agents ( stt llm tts pipeline running) we have 5 agent nodes c7a.xlarge network loadbalncer distrubute traffic in each agent node i have ideal workers set to 10 no i need to run 50 concurrent calls so i made 5 agent nodes will it work is any chnages reuqire to agent or livrkit to support horizonatl scaling

Thanks for reaching out, Bhavya. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i just need to test so not doing auto scaling just already 5 instance under nlb running

Thanks — for a test, your 5 agent instances behind the NLB (each with ideal_workers = 10) should support ~50 concurrent calls. Quick checklist to ensure the test runs smoothly: • Monitor LiveKit server (t2.large) CPU/memory and Redis — they can become bottlenecks even if agents are fine. • Ensure each agent can actually handle 10 concurrent pipelines (STT/LLM/TTS) under your workload. • Configure NLB health checks and set long enough connection/idle timeouts so websocket connections remain stable. • Use graceful shutdown/drain on agents if you redeploy so active calls aren’t dropped. • Watch metrics during the test: per-node active sessions, CPU, memory, and any dropped/rejected jobs. If you want, share a short sample of observed CPU/memory during a few concurrent calls and I’ll confirm capacity or suggest instance tweaks. , with interactive elements

will network laodbalncer distrubute calls

Good question — short answer: the NLB will distribute incoming TCP/WebSocket connections across the 5 agent instances, but LiveKit Server ultimately assigns each call to an agent based on agent availability/load. Details to keep in mind: • NLB role: it balances raw network connections (TCP/WebSocket) and runs health checks. It does not make job assignment decisions. • LiveKit role: LiveKit Server routes agent jobs (sessions) to available agents using its load function and

load_threshold

. That is what determines which agent handles a call. • What to check for your test: ◦ Ensure the NLB health checks hit the correct agent endpoint and that idle/connection timeouts are long enough for WebSocket sessions. ◦ Agents must be able to connect/register to the LiveKit Server and Redis (outbound WebSocket OK). ◦ Verify agents expose the correct ports and pass health checks so they’re eligible for traffic. ◦ Monitor per-node active sessions and CPU/memory to confirm calls are actually distributed and nodes aren’t saturated. If you want, share your NLB health-check path/config and agent registration logs and I’ll verify they’re correct for the test. , with interactive elements

it failed i placed calls 50 calls didnt worked

Sorry that happened — I’ll help troubleshoot. A few likely causes and quick checks so we can narrow it down: What I’m responding to: your 50-call test failed with 5 agent instances behind the NLB. Immediate checks (please share results or logs for any that fail): • LiveKit server & Redis: check CPU, memory, and Redis command latency during the test. Did the t2.large spike or Redis show slow ops? • Agent process load: on each agent node, check actual active sessions, CPU, and memory. Did any node exceed ~70% CPU or OOM? • Agent logs: any errors like “room connection not established within 10 seconds” or job rejection messages? • LiveKit server logs: any errors assigning jobs or connection timeouts? • NLB health checks and timeouts: confirm health check path, and ensure idle/connection timeouts are long enough for WebSocket (keepalive). • Agent registration: confirm all agents are connected/registered with the LiveKit server during the test. • Worker config: verify

ideal_workers = 10

is actually applied and that the agent’s `load_threshold`/`load_fnc` haven’t made nodes appear unavailable. If you can, send: • One agent log excerpt around the failure time • LiveKit server error lines around the same time • CPU/memory numbers for LiveKit and one agent node during the test • NLB health-check config (protocol, path, interval, timeout) With those I’ll point to the exact root cause and give concrete fixes (instance sizing, config adjustments, or job routing fixes). Sources: Self-hosted deployments | LiveKit Documentation | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Why WebRTC beats WebSockets for realtime voice AI | LiveKit | Pricing | LiveKit, with interactive elements

AGENT_IDLE_PROCESSES=25 on 4 cpu instance its not making 25 concurrent calls

Thanks — I’m responding to your note that you set

AGENT_IDLE_PROCESSES=25

on a 4‑vCPU instance but it’s not handling 25 concurrent calls. Short answer:

AGENT_IDLE_PROCESSES

only sets the maximum/minimum worker count; actual concurrent call capacity is limited by CPU, memory, and the resource cost of your STT/LLM/TTS pipeline. A 4‑vCPU instance cannot reliably run 25 concurrent voice pipelines. What to do next: • Reduce per‑node workers to a realistic value. For a 4‑vCPU / 8GB node, try 8–12 workers and test load. • Scale out instead of packing more workers on one host: add more agent nodes or use larger instance types (more vCPU/RAM) if you must increase per‑node concurrency. • Tune agent load controls: ensure

load_fnc

and

load_threshold

reflect CPU usage so LiveKit marks nodes unavailable before they overload. • Monitor during tests: per‑node CPU, memory, process counts, and agent logs (look for job rejection or timeout messages). • If you share one agent’s CPU/memory and a sample agent log during failures, I’ll recommend the exact worker count and instance size. If you want, I can suggest specific worker counts for common instance types based on your pipeline (STT local vs external, LLM CPU usage, TTS cost) — tell me which components run locally. , with interactive elements

what is BRIDGE_WORKERS=4

You’re asking what

BRIDGE_WORKERS=4

means. Brief answer: it sets the number of worker threads the LiveKit server uses for SFU media processing (packet forwarding / media pipeline).

BRIDGE_WORKERS=4

means LiveKit will run 4 bridge workers to process/route audio/video. Notes relevant to your setup: • This is for the LiveKit server process, not the agent processes. It does not increase agent-side concurrent calls. • Best practice: set

BRIDGE_WORKERS

roughly to the number of CPU cores on the machine running the LiveKit server for best throughput. • For your 50-call test, increasing

BRIDGE_WORKERS

on the t2.large (limited CPU) won’t by itself allow more agent sessions — you still need enough LiveKit/Redis capacity and properly sized agent nodes. Sources: Going beyond a single-core | LiveKit, with interactive elements

if we move to eks for 50 calls what setup we need and how cost monhtly

Thanks — you asked what’s needed to run 50 concurrent calls on EKS and how much it might cost monthly. Below is a concise, practical plan and cost guidance to run a reliable test/production deployment. Recommended EKS architecture (for 50 concurrent voice-AI calls) • LiveKit server ◦ Run as 1–2 replicas (for HA) on dedicated nodes. ◦ Set

BRIDGE_WORKERS

≈ CPU cores per LiveKit server node (match vCPU count). ◦ Use shared Redis for signaling/coordination. ◦ Ensure readiness/health probes and

terminationGracePeriodSeconds

for graceful shutdown. • Agent deployment (stateful workers) ◦ Containerize your agent (STT/LLM/TTS) and run as Deployment/ReplicaSet. ◦ Use Pod resources and requests/limits so Kube schedules predictably. ◦ Set

AGENT_IDLE_PROCESSES

/ worker count per pod to a realistic value (for 4 vCPU / 8GB pod, use ~8–12 workers — not 25). ◦ Use Horizontal Pod Autoscaler (HPA) based on CPU (or custom metric that matches

load_fnc

) and scale-out before pods hit

load_threshold

(e.g., scale at 50% if

load_threshold

= 70%). ◦ Set lifecycle preStop to drain and give long termination grace (10+ minutes) for active calls. • Node sizing and counts (starting point) ◦ Agent nodes: use instances with ~4 vCPU / 8GB each (e.g., c7a.xlarge). To hit ~50 concurrent calls: ▪︎ If you place ~10 workers per node → 5 agent pods/nodes (same as your current plan). ▪︎ Option: run multiple pods per node if pod sizing allows; prefer uniform sizing and let HPA scale pods. ◦ LiveKit server + Redis: run on higher-capacity instances (4+ vCPU) or separate small node pool. • Networking ◦ Use NLB for ingress to LiveKit clients (WebRTC signaling). Agents connect outbound to LiveKit — no public inbound needed for agents. ◦ Configure NLB idle timeout and health checks appropriate for WebSocket/long-lived connections. Monitoring, observability & tuning • Monitor per-pod CPU, memory, active sessions, and job rejection/timeouts. • Log errors like “room connection not established within 10 seconds” — that indicates insufficient available workers. • Tune

load_fnc

and

load_threshold

on agents so LiveKit sees accurate availability. Rough cost guidance (monthly, ballpark — depends on region/pricing and run-time) • Example baseline (on-demand, us-east-1 style pricing approximation): ◦ 5 agent nodes (c7a.xlarge or equivalent 4 vCPU/8GB) ~ 5 × ~$60–80/mo = $300–400/mo ◦ 1–2 LiveKit server nodes (4 vCPU) ~ 1–2 × ~$60–80/mo = $60–160/mo ◦ 1 Redis (small) ~ ~$30–60/mo ◦ EKS control plane + networking + EBS storage ~ ~$50–150/mo ◦ Total rough range: $440–770/mo (minimal on-demand estimate). Notes: this is a rough estimate — actual cost varies by instance family, reserved/spot vs on-demand, region, storage IO, data egress, and external model API costs (STT/LLM/TTS providers). If you want I can: • Propose exact Kubernetes manifests (resource requests/limits, probes, HPA settings, preStop hook) for your agent and LiveKit server. • Refine cost estimate if you tell me region, whether you’ll use on-demand/spot/reserved, and whether STT/LLM/TTS run locally or use external APIs. Sources: Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | Self-hosted deployments | LiveKit Documentation, with interactive elements