Thanks — I’m responding to your request for a recommended self‑hosted LiveKit infrastructure to support 200 concurrent one‑on‑one voice sessions (1 local AI agent + 1 remote participant with an audio track per room). Below is a concise, practical plan you can use to size and deploy the environment. High-level recommendation • Use a small LiveKit SFU cluster for media routing and a horizontally scaled pool of agent worker nodes for the AI sessions. • Run everything on Kubernetes (or another container orchestrator) so you can autoscale workers and manage rollouts, monitoring, and configs centrally. Sizing guidance (starting point) • Agent worker nodes (handles STT, LLM context, TTS, and the agent process): ◦ Per node: 4 vCPU, 8 GB RAM, ~10 GB ephemeral storage ◦ Concurrency per node: ~10–25 concurrent agent sessions (depends on STT/LLM/TTS CPU and memory needs) ◦ To reach 200 sessions: provision ~8–20 agent nodes (start with 12 nodes as a balanced starting point, then scale by measured utilization) • LiveKit SFU cluster (media forwarding): ◦ Audio-only, one-to-one calls do not require transcoding; SFU CPU load is relatively low per session. ◦ Start with 2–3 SFU pods behind a load balancer for redundancy and capacity; scale horizontally if CPU/network becomes constrained during load testing. • Network and disk: ◦ Ensure high network throughput and low latency between SFU and agent nodes (same VPC/region). ◦ Agent nodes: ~10 GB ephemeral storage for models/logs; persist logs centrally (ELK/Cloud logging). • Other infra: ◦ Signaling/turn servers: run TURN if clients are behind restrictive NATs; one or more TURN instances sized for concurrent relayed streams. ◦ Redis or similar for transient state if your agent system requires shared state/queues. Operational & scaling guidance • Autoscale agent workers by CPU and request queue length (not just number of pods). Use HPA based on CPU + a custom metric for inference concurrency. • Scale SFU pods horizontally based on network throughput and CPU (monitor per-pod connection count and network TX/RX). • Run load tests that mimic real users (cloud VMs, gradual ramp, realistic audio traffic). • Tune OS/network limits on nodes (file descriptors, UDP buffer sizes) before full load testing. Monitoring and metrics to track • CPU, memory, and GPU (if used) per agent node. • Connection counts and join/fail rates on SFU pods. • STT/TTS latency, LLM inference latency, and end-to-end join + first-audio time. • Network throughput per SFU pod and per node. • File descriptor/socket usage. Failure domains and redundancy • Spread SFU and agent pods across multiple availability zones. • Ensure at least 2 SFU pods for failover. • Use readiness/liveness probes and graceful shutdown to avoid dropped calls during rolling updates. How to proceed (practical next steps) 1. Start with a small deployment: a. Deploy 12 agent nodes (4 vCPU / 8 GB each) and 3 SFU pods. b. Configure TURN and a small upstream STT/TTS/LLM setup matching expected production models. 1. Run an incremental load test up to 200 concurrent sessions, monitoring CPU, memory, network, join latency, and STT/TTS latency. 2. Adjust agent node count and SFU replicas based on observed sessions-per-node and media throughput. If per-node concurrency is lower than expected, add more agent nodes. 3. Configure autoscaling rules after you identify realistic thresholds. If you’d like, I can: • Help estimate costs for a specific cloud provider and instance types. • Suggest concrete HPA metrics and Kubernetes manifests for autoscaling SFU and agent pools. • Propose a testing ramp plan (number of steps, duration, traffic profile) to validate this setup. Would you like me to draft a sample Kubernetes manifest and autoscaling rules for the starting configuration (12 agent nodes, 3 SFU pods), or estimate costs for AWS/GCP instance types? Sources: Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Self-hosted deployments | LiveKit Documentation | Why WebRTC beats WebSockets for realtime voice AI | LiveKit | Bringing Zoom's end-to-end optimizations to WebRTC | LiveKit | How to load test voice agents and get meaningful results | LiveKit, with interactive elements