Hey folks 👋 My Startup is Running a Voice AI setup on a single EC2 (c7i.4xlarge – 16 vCPU, 32GB) with LiveKit + FastAPI + Twilio + STT→LLM→TTS + Redis. Trying to hit 50+ concurrent calls, but it starts failing around 10–25 calls. Looking for advice on: • Best way to scale this on AWS (ECS / EKS / multi-node?) ( we need to save cost also) • How to scale LiveKit + real-time pipelines properly • Any known bottlenecks in this kind of setup Would really appreciate any pointers Load testing we found for 20+ users concurrent ( 25 % cpu spikes ) Latency Bottleneck (LLM vs TTS vs STT/other) • LLM TTFB: count 161, avg 624.6ms, p95 1359ms, max 2683ms. • TTS TTFB: count 178, avg 749.2ms, p95 1347ms, max 1679ms. • End-to-end response latency: count 31, avg 1576.4ms, p95 3113.5ms, max 3301ms. • Estimated non-LLM/TTS overhead: about 202.6ms on average.

Thanks for reaching out, Bhavya. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

using external APIs, another thing cannot go with eks as we are early startup need low cost solution curent target is 50+ concurrent calls

Thanks — great clarification. Since you prefer a low‑cost ECS-based approach and use external LLM/TTS/STT APIs, here’s a focused, actionable plan to reach 50+ concurrent calls with cost control. Summary of approach • Use ECS (Fargate for simplicity or ECS on EC2 Spot for lower cost) and horizontally scale LiveKit + agent workers across multiple nodes. Keep AI inference external (APIs), optimize streaming, and split responsibilities to avoid CPU hotspots. Concrete recommendations 1) Architecture (high level) • LiveKit servers (stateless signaling + media relay) deployed as multiple ECS services behind a Network Load Balancer. • Agent workers (FastAPI + per-session agent) as a separate ECS service with autoscaling. • External LLM/TTS/STT remain as API calls; place agent workers and LiveKit in same AZ as NAT/egress to reduce RTT. • Redis (session/locks) on ElastiCache in same AZ; enable connection pooling. 2) ECS mode choices (pick one) • ECS Fargate: ◦ Pros: minimal ops, predictable pricing, scales easily. ◦ Cons: slightly higher cost vs spot EC2. • ECS on EC2 with Spot instances: ◦ Pros: cheapest at scale if you can handle occasional interruptions. ◦ Use a mixed-instance Auto Scaling Group (e.g., c6i/c7i or c5/c6g) with lifecycle hooks and graceful draining. 3) Node sizing & initial capacity (starting point for 50 concurrent) • LiveKit server tasks: 2–3 tasks on t3.medium / t3.large (or Fargate 0.5–1 vCPU / 1–2 GB) for signaling; increase if media forwarding CPU rises. • Agent worker tasks: estimate ~0.5–1 vCPU per concurrent session depending on CPU work (VAD/STT preproc). For safety start with: ◦ If Fargate: 50 concurrent → 25 tasks with 1 vCPU / 2GB (each handles ~2 sessions with lightweight work) OR scale to 50 tasks with 0.5 vCPU if your code supports it. ◦ If EC2 Spot: use many smaller instances (e.g., c7i.large or c6i.large) to pack workers and reduce wasted CPU. • Redis: cache nodes medium (e.g., cache.t3.medium) and scale up if latency/ops are high. 4) Autoscaling policy suggestions • Agent service autoscale target: ◦ Metric: CPU utilization target 50–60% OR custom metric = active sessions per task. ◦ Scale out: add tasks when CPU > 60% for 60s or when queue depth > X. ◦ Scale in: when CPU < 30% for 300s with cooldown. • LiveKit tasks autoscale: ◦ Metric: CPU > 60% or high connection count per task. • Use ECS Service Autoscaling + Application Auto Scaling with step policies. 5) Latency optimizations (APIs + streaming) • Use streaming STT/LLM if the external provider supports it to overlap processing and TTS. • Start TTS on partial LLM output where possible to reduce end-to-end TTFB. • Keep API requests from agent workers pooled and reuse connections to reduce TLS handshake time. • Co-locate NAT/Gateway and use ENIs in same AZ to reduce extra hops. 6) Cost controls • Use EC2 Spot for agent-heavy nodes; keep LiveKit servers on on-demand or small Fargate tasks for reliability. • Right-size task CPU/memory and monitor to avoid overprovisioning. • Use autoscaling with aggressive scale-in cooldowns during low traffic. • Cache frequent responses (TTS audio or LLM answers) for repeated questions to avoid API costs and latency. 7) Observability & load testing • Track per-stage metrics: STT TTFB, LLM TTFB, TTS TTFB, CPU per task, Redis latency, network egress. • Re-run load tests after: ◦ separating inference traffic, ◦ enabling streaming, ◦ scaling workers to confirm where failures occur. • Use these metrics to tune CPU targets and task counts. Quick action checklist (first steps) 1. Containerize agent + FastAPI and deploy to ECS (Fargate) with 0.5–1 vCPU tasks. 2. Deploy 2–3 LiveKit tasks behind an NLB. 3. Configure agent autoscaling on CPU (target 50–60%) and test up to 50 concurrent calls. 4. Enable streaming with external STT/LLM/TTS where possible and measure end-to-end latency improvements. 5. If costs too high, migrate agent tasks to ECS on Spot with mixed-instance groups. If you want, I can: • propose exact ECS task sizes and numbers for Fargate vs Spot EC2 with an estimated hourly cost, • or produce an ECS autoscaling policy config (CPU thresholds and step sizes) you can drop into your deployment. Which of those would help most next? , with interactive elements

we need to save cost thats why we were using c7i.4xlarge ec2 ec2 is cheaper than ec2

Got it — you prefer staying on EC2 for lower cost. Here’s a concise, cost-optimized plan to reach 50+ concurrent calls using multi-node EC2 (Spot + On‑Demand), LiveKit, and external AI APIs. Summary • Keep LiveKit + agents on EC2 with a mixed Auto Scaling Group: spot instances for agent-heavy work, a small set of on‑demand instances for reliability (LiveKit servers). • Horizontally scale workers across many smaller instances rather than one large instance. • Focus on streaming external APIs, co‑location, and autoscaling on CPU/active sessions to control cost and hit your 50+ target. Concrete architecture • LiveKit servers (signaling/media): 2–3 on-demand EC2 instances (e.g., c7i.large or c6i.large) behind a Network Load Balancer. These handle signaling and media relay and should be reliable. • Agent worker fleet (FastAPI + per-session agent): Auto Scaling Group with mixed instances using Spot as primary and On‑Demand fallback. Use smaller compute instances (e.g., c7i.large or c6i.large) to pack many agents. • Redis: ElastiCache (clustered or single AZ depending on budget) in same AZ. • NAT/Egress and VPC endpoints configured so agent workers and LiveKit are in same AZ to minimize RTT to external APIs. Sizing guidance (starting point) • Aim to spread sessions: target ~2–4 concurrent sessions per c7i.large instance initially (adjust after testing). • For 50 concurrent sessions, start with ~15–25 c7i.large Spot instances (depends on per-session CPU). If each c7i.large can reliably handle 3 sessions, need ~17 instances. • Keep 2–3 on‑demand c7i.large (or slightly larger) for LiveKit and to absorb spot interruptions. Autoscaling & reliability • Use an Auto Scaling Group with mixed instances (multiple instance types) and Spot allocation strategy (capacity-optimized). • Lifecycle hooks and graceful draining: ensure sessions can reassign or reconnect on instance termination. • Scale triggers: ◦ Primary: CPU target ~50–60% (scale out when >60% for 60s). ◦ Secondary: custom metric — active sessions per instance or task queue depth. • Maintain a small On‑Demand capacity buffer (e.g., 10–20% of desired instances) to avoid sudden capacity loss if Spot is reclaimed. Cost controls • Use Spot for agent workers with capacity-optimized allocation and diversified instance types (c6i, c7i, c5). This reduces cost dramatically. • Keep LiveKit servers on small on‑demand instances for stability. • Right-size by measuring actual sessions-per-instance in load tests; reduce instance size if CPU is low. • Cache frequent LLM/TTS responses to cut API calls. • Use connection pooling and HTTP keep-alive to cut API/TLS overhead and egress time (reduces cost from repeated handshakes). Latency & throughput optimizations • Use streaming STT/LLM/TTS APIs to overlap stages and reduce end-to-end latency. • Co-locate instances in same AZ and region as your NAT/gateway to reduce RTT to external APIs. • Parallelize independent work (e.g., start TTS as LLM partials arrive). • Monitor Redis latency and tune connection pool sizes. Operational checklist (first actions) 1. Create ASG with mixed Spot/On‑Demand, capacity-optimized strategy and at least two AZs. 2. Deploy 2–3 on‑demand LiveKit servers behind NLB; open required ports. 3. Launch agent AMI/container on Spot instances with graceful drain hooks. 4. Run load tests incrementally to find sessions-per-instance, then tune ASG target capacity and CPU thresholds. 5. Enable metrics: CPU, network, Redis latency, per-stage TTFB (STT/LLM/TTS). If you want, I can: • estimate hourly cost for a Spot-heavy fleet vs single c7i.4xlarge, • or propose an exact ASG instance mix (types and counts) to start load testing immediately. Which would you like next? , with interactive elements

bro can you please simplify

Totally — here’s a simple, actionable plan to get 50+ concurrent calls on EC2 while keeping costs low. Big idea • Stop running everything on one large box. Run many smaller EC2s (mostly Spot) for workers, and a few on‑demand instances for LiveKit/signaling. Autoscale by CPU/active sessions. 3-step plan (do this first) 1) Set up instances • LiveKit (reliable): 2–3 on‑demand c7i.large (or c6i.large) behind a Network Load Balancer. • Workers (cost-sensitive): Auto Scaling Group with Spot c7i.large / c6i.large instances. Start with ~15 instances and adjust after tests. • Redis: ElastiCache (same AZ). 2) Autoscale rules • Primary signal: CPU. Scale out when avg CPU > 60% for 60s; scale in when CPU < 30% for 300s. • Add a custom metric later (active sessions per instance) if you can measure it. • Keep a small on‑demand buffer (10–20%) so spot reclaim doesn’t kill capacity. 3) Latency & cost optimizations • Use streaming STT/LLM/TTS to overlap work and cut end-to-end time. • Co-locate everything in the same AZ to lower RTT to external APIs. • Reuse HTTP/TLS connections (keep‑alive) to reduce API overhead. • Cache common LLM/TTS replies to save API calls. • Use spot for workers; keep LiveKit on on‑demand. Quick monitoring targets • Aim initially: ~3 sessions per c7i.large. Verify with load test; adjust instance count. • Watch: CPU, Redis latency, STT/LLM/TTS TTFB, and end-to-end p95. If you want, I’ll: • give an exact starting ASG mix (instance types + counts) with an estimated hourly cost, or • produce the autoscaling rules/CloudFormation snippet you can drop in. Which one should I do? , with interactive elements

We are on dev we need simple setup for voice calls 50 + concurrent with low cost infra

Perfect — here’s a minimal, developer-friendly EC2 setup that’s cheap, simple, and aimed at 50+ concurrent voice calls. One‑line architecture • A few on‑demand EC2s for LiveKit (signaling/media) + an Auto Scaling Group of smaller Spot EC2s for worker processes (FastAPI agents); Redis on ElastiCache; keep everything in the same AZ. Concrete starter config (drop-in) 1. LiveKit (reliable) ◦ 2 on‑demand instances: c7i.large (2 vCPU, 4–8 GB) behind a Network Load Balancer. ◦ Roles: signaling, media relay, health checks enabled. 1. Worker fleet (cost‑optimized) ◦ Auto Scaling Group (mixed instances) using Spot as primary, on‑demand fallback. ◦ Instance types: c7i.large or c6i.large. ◦ Start count: 15 Spot instances (adjust after tests). ◦ Expected packing: assume ~2–4 concurrent sessions per instance initially. 1. Redis ◦ ElastiCache (single AZ) t3.medium to start; colocate in same AZ. Autoscaling rules (easy, effective) • Scale out: when average CPU > 60% for 60s — add +2 instances. • Scale in: when average CPU < 30% for 300s — remove −1 instance. • Keep a 10–20% on‑demand capacity buffer in ASG to survive Spot reclaims. Networking & reliability • Put LiveKit + workers in same AZ to minimize RTT to external APIs. • Open required ports for LiveKit (WSS/HTTPS + UDP media). • Use lifecycle hooks and graceful shutdown scripts on worker instances so sessions can reconnect. Latency & cost quick wins • Use streaming STT/LLM/TTS where available to overlap work. • Reuse HTTP/TLS connections (keep‑alive) to external APIs. • Cache common responses (TTS audio) to cut API calls and cost. • Tune per-instance concurrency after one load test pass (measure sessions-per-instance and adjust instance count). How to validate fast 1. Deploy LiveKit (2 on‑demand) + 5 workers first; run incremental load tests up to 50. 2. Observe CPU and sessions-per-instance; scale ASG target to hit target concurrency. 3. Tune worker count and Spot mix based on real session density. If you want, I’ll provide: • Exact ASG instance mix (types + counts) with a rough hourly cost estimate, or • A simple shutdown/graceful‑drain script snippet for the workers. Which would you like next? , with interactive elements