Is it possible to scale livekit server on AWS fargate?

Thanks for reaching out, Haraprasad. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

yes I want to experiment with fargate, I want to go serverless because of costing and irregular spikes and most of the time idle to no uses other than the spikes

Thanks — that’s helpful context. I’ll walk you through a practical checklist and the key technical considerations for experimenting with LiveKit Server on AWS Fargate, focused on cost-efficiency for bursty traffic. I’ll also ask one quick follow-up so I can tailor the checklist to your expected load. Quick follow-up • Approximately how many concurrent participants (or concurrent peer connections) do you expect at peak? Even a rough number (e.g., 50, 500, 2,000) helps recommend CPU/memory and autoscaling triggers. Important note up front • Fargate is not a documented or tested deployment target for LiveKit. You can experiment, but expect to validate networking (UDP), port mapping, and scaling behavior. If anything fails, the supported alternatives are self-hosted Kubernetes (Helm), EC2, or LiveKit Cloud. Key technical considerations and checklist for experimenting on Fargate 1) Networking & UDP • LiveKit uses WebRTC which relies on UDP for media (SRTP/DTLS). Confirm your Fargate setup can handle UDP traffic end-to-end. • Use an NLB (Network Load Balancer) that supports UDP to front your tasks. ALB is HTTP(S) — it won’t forward raw UDP. • Ensure Fargate tasks are attached to awsvpc networking mode so tasks get ENIs and routable IPs. 2) NAT, public IPs & ICE candidates • Clients need ICE candidates that resolve to reachable IPs/ports. If tasks are in private subnets behind NAT, ensure the public-facing NLB forwards correctly and that ICE candidates reflect reachable addresses. • Consider using TURN servers (coturn or hosted) for reliable NAT traversal. Ensure TURN is reachable from clients and from LiveKit instances. 3) Ports and task definition • Expose and map the necessary ports in the Fargate task: ◦ WebSocket/HTTP(s) API port (typically 7880/7881 or your configured port) ◦ UDP media ports (range configured in LiveKit e.g., 40000-40100) — ensure NLB and security groups allow this range. • Fargate requires explicit port mappings and SECG rules. Verify UDP rules are present. 4) Health checks & lifecycle • Use application-level health checks (HTTP) so the load balancer only routes traffic to healthy tasks. • LiveKit maintains state per room/participant on the server. If a task is terminated, sessions tied to that process will be interrupted. Plan for session reconnect behavior at the client. • Consider sticky routing or a coordination layer if you need participants in the same room to hit the same process; otherwise rooms may be spread across instances and require shared room management (LiveKit server instances can coordinate, but you must verify behavior in your test). 5) Storage & state • LiveKit instances are typically ephemeral. If you need durable recording, persistent storage, or shared state, configure external services (S3 for recordings, Redis or DB for signalling/state if used). • Confirm how autoscaling/termination affects ongoing recordings or data. 6) Autoscaling strategy • Use Fargate Service Auto Scaling with CloudWatch metrics. Possible autoscaling signals: ◦ CPU or memory utilization of tasks ◦ Custom metrics: active peer count, active tracks, or aggregated websocket connections published to CloudWatch via a metrics exporter • Because WebRTC sessions may persist, scale-up should be proactive (based on connection rate or queue) and scale-down should be conservative to avoid dropping active sessions. 7) Cost trade-offs • Fargate avoids node management but each running task has cost even when idle. Autoscaling down to zero can save costs but note cold-start delays and session interruptions. • Evaluate whether quickly spinning tasks up during spikes meets your latency/UX requirements. 8) Observability & logging • Push LiveKit logs and metrics to CloudWatch or a centralized system for troubleshooting (connections, ICE failures, packet loss). • Monitor UDP packet loss, connection establishment times, and TURN usage. 9) Security & configuration • Use environment variables or AWS Secrets Manager for keys and credentials (API key/secret). • Secure the API endpoint (TLS) and set appropriate CORS/ACLs. 10) Test plan (suggested steps) 1. Start with a minimal proof-of-concept: ◦ Run 1 task behind an NLB, verify clients can connect, gather ICE candidates, and exchange media. 1. Add TURN and test NAT traversal from diverse networks (mobile, corporate NAT). 2. Scale to a small cluster of tasks and verify that multiple tasks can run concurrently; validate room assignment and reconnects. 3. Simulate spikes (connection bursts) and observe scaling behavior and cold-start times. 4. Test task termination during active sessions to observe client reconnect behavior and recovery. What I can provide next • A checklist of exact LiveKit config fields you’ll want to set for UDP port ranges, TURN settings, and API binding, if you want the concrete config examples. • A suggested CloudFormation/ECS task definition snippet and NLB setup outline for UDP forwarding. • Guidance on metrics to export (how to count active peers) and autoscaling rules. If you share the expected peak concurrent participants and whether you already have a TURN server or plan to use a hosted TURN, I’ll produce a tailored config checklist and a sample Fargate task/NLB setup to try first.

1000-2000 users

Thanks — that helps a lot. I’ll respond to your peak target of 1000–2000 concurrent users and focus on practical guidance for experimenting with LiveKit Server on AWS Fargate. Remember: Fargate is not a documented/tested deployment for LiveKit, so you’ll need to validate these steps in a PoC. I’ll give you a concise, actionable checklist, sizing guidance, autoscaling ideas, and a sample ECS/Fargate task-level outline you can use to start. High-level approach • Build a small proof-of-concept first (one task behind an NLB) to validate UDP media flow, ICE candidates, and TURN behavior. • Iterate to a load test that simulates your expected peak, then tune autoscaling, cold-start, and capacity planning. Quick assumptions I’m using for recommendations • “Concurrent users” = concurrent peer connections (clients connected to the LiveKit server). • Live media uses UDP (WebRTC), and TURN may be required for NAT traversal. • You want cost-efficient autoscaling with the ability to scale toward zero when idle. 1) Sizing guidance (starting points) • Start conservative for the PoC, then scale horizontally: ◦ PoC task: 2 vCPU / 4 GB RAM — verify connectivity and basic media. ◦ Small-scale test node: 4 vCPU / 8–16 GB RAM — use for load tests. • For 1,000–2,000 concurrent users you will likely need multiple tasks. Exact per-task capacity depends on number of tracks per user, bitrate, and codec usage. Measure: ◦ CPU per active sender/receiver ◦ Memory growth with concurrent tracks • Plan to scale horizontally (multiple tasks) rather than a single large task. 2) Networking and load balancing • Use awsvpc networking mode so each task receives an ENI and routable IP. • Use a Network Load Balancer (NLB) that supports UDP to forward media ports to tasks. • Ensure Security Groups and NACLs allow the UDP port range used by LiveKit. • Expose: ◦ TCP port for API/WS (e.g., 7880/7881 or your configured API port) via NLB target group using TCP/HTTP as appropriate. ◦ UDP port range for RTP/SRTP (choose a contiguous range, e.g., 40000–40100) via NLB UDP listeners. • Verify public ICE candidates resolve to the NLB address and that clients receive correct candidates. 3) TURN and NAT traversal • TURN is critical for reliability across diverse client networks. If you don’t already have TURN: ◦ Run a coturn cluster (on EC2 or other) or use a hosted TURN provider. ◦ Ensure TURN is reachable and allowed in security groups from clients and tasks. • Configure LiveKit to use the TURN server(s) so clients can connect even with restrictive NATs. 4) Ports and LiveKit configuration • Configure LiveKit server to bind to the UDP port range you choose and to advertise the public IP/hostname (the NLB address or a managed DNS pointing to it) so ICE candidates are reachable. • Use environment variables or secrets for API key/secret configuration. • Keep recordings, if used, in S3 or external storage rather than local disk. 5) Autoscaling strategy • Scale based on custom metrics rather than only CPU: ◦ Active peer count or websocket connection count aggregated to CloudWatch (expose LiveKit metrics or use a sidecar exporter). ◦ Connection creation rate (to scale up proactively during bursts). • Use step scaling with conservative cooldowns for scale-down to avoid dropping active sessions. • Consider a minimum number of tasks > 0 if cold-start times are unacceptable for UX; otherwise allow scaling to zero to save costs and accept cold start delays. 6) Session resilience & client behavior • LiveKit instances hold session state. Expect client reconnects when tasks are replaced. • Implement client reconnect/backoff logic and session resume where possible. • Test behavior when a task is terminated mid-session to validate user impact. 7) Observability & testing • Export metrics (active peers, tracks, packet loss) to CloudWatch or Prometheus-compatible endpoints. • Centralize logs to CloudWatch Logs for troubleshooting. • Load test with realistic scenarios (number of tracks, bitrate, connection churn) to derive per-task capacity numbers. 8) Cost and cold-start considerations • Fargate removes node management but still incurs cost for running tasks. • Autoscaling to zero saves cost but causes cold starts; measure cold-start latency to see if it fits your UX. • Compare total cost with small EC2 Auto Scaling group or Kubernetes cluster that can use burstable instances if cold-starts or UDP behavior aren’t acceptable. 9) Suggested PoC testing steps 1. Deploy 1 task behind an NLB (UDP + TCP listeners); verify WebSocket and media flow from multiple client networks. 2. Add TURN and validate across NAT types (mobile, corporate). 3. Instrument LiveKit to push connection/peer metrics to CloudWatch. 4. Deploy a small fleet (3–5 tasks), run load tests, and monitor metrics. 5. Implement and tune autoscaling rules based on the observed metrics. 10) ECS/Fargate task outline (conceptual) • Task definition: ◦ Network mode:

awsvpc

◦ CPU/Memory: start with

2048

CPU units (2 vCPU) and

4096

MiB for PoC; increase for load tests ◦ Container port mappings: ▪︎ TCP: API/WS port (e.g., 7880) ▪︎ UDP: media port range (e.g., 40000–40100) — map these as container ports in the task and ensure NLB forwards UDP ◦ Environment variables / secrets: ▪︎ LIVEKIT_API_KEY, LIVEKIT_API_SECRET (use Secrets Manager) ▪︎ LIVEKIT_UDP_PORT_MIN, LIVEKIT_UDP_PORT_MAX (match your chosen range) ▪︎ LIVEKIT_ANNOUNCED_IP / LIVEKIT_ANNOUNCED_DOMAIN (public NLB DNS or IP) ▪︎ TURN URLs / credentials • Service: ◦ Attach to NLB target groups (one for TCP/WS, one for UDP range) ◦ Health check: HTTP health endpoint of LiveKit ◦ Autoscaling: target custom metric or CloudWatch metric (see above) If you’d like, I can produce: • A sample ECS task definition JSON snippet and abbreviated CloudFormation outline for the NLB + Fargate service (UDP + TCP listeners). • A short list of exact LiveKit environment variables to configure for UDP port ranges and announced IPs. • A recommended set of metrics and CloudWatch alarms to feed autoscaling. Next question • Do you already have a TURN server (self-hosted or hosted)? If not, do you want a sample coturn setup and configuration to use with LiveKit? Tell me which of the three deliverables above you’d like next (task definition snippet, LiveKit env var list, or metrics/autoscaling rules), and confirm whether you have TURN — I’ll generate the concrete artifacts.

No Turn now, simple setup for poc proving scalability upto 1000 and cost effective

Thanks — that’s clear. Below is a concise, practical PoC guide to run LiveKit on AWS Fargate without TURN, aimed at proving scalability up to ~1000 concurrent users while keeping costs sensible. Remember: Fargate is not a documented/tested deployment for LiveKit, so treat this as an experimental PoC and validate behavior (especially UDP/ICE) during testing. PoC goals • Validate UDP media flow and ICE candidate reachability without TURN. • Demonstrate scaling to ~1000 concurrent users. • Keep architecture simple and cost-conscious. High-level architecture • Fargate Service (awsvpc) running LiveKit container(s) • Network Load Balancer (NLB) in front for UDP + TCP • CloudWatch metrics for autoscaling (custom metric for active peers) • S3 for recordings (if used); no persistent local storage • No TURN for PoC (expect some clients behind restrictive NATs may fail — test from various networks) Recommended minimal settings for PoC • LiveKit container (start): ◦ CPU: 2 vCPU (2048 CPU units) ◦ Memory: 4 GB • LiveKit container (for load testing / capacity baseline): ◦ CPU: 4 vCPU ◦ Memory: 8–16 GB • UDP port range: 40000–40100 (100 ports) — configure both LiveKit and NLB to use this range • TCP/WS port: 7880 (or your configured API/WS port) • Network mode: awsvpc (each task gets an ENI) Key steps — deploy PoC 1) Create NLB with listeners • TCP listener for API/WS (port 7880 -> target group for your tasks using TCP). • UDP listener(s) for media port range (create a UDP listener or multiple listeners covering the port range) forwarding to target group(s) with UDP protocol. • Use cross-zone load balancing and target groups mapped to the Fargate service. 2) Define Fargate task (conceptual) • Network mode:

awsvpc

• Container port mappings: ◦ TCP 7880 ◦ UDP 40000–40100 (expose as container ports) • Environment variables (example names): ◦ LIVEKIT_API_KEY / LIVEKIT_API_SECRET (store in Secrets Manager) ◦ LIVEKIT_PORT (or equivalent) for API ◦ LIVEKIT_UDP_PORT_MIN=40000 ◦ LIVEKIT_UDP_PORT_MAX=40100 ◦ LIVEKIT_ANNOUNCED_IP or LIVEKIT_ANNOUNCED_DOMAIN = NLB public IP or DNS • Health check: HTTP health endpoint of LiveKit 3) Security groups & subnets • Place tasks in private subnets with NAT for outbound (or public subnets if you prefer tasks with public IPs). • Security groups: allow inbound TCP 7880 and UDP 40000–40100 from 0.0.0.0/0 (or restrict to your client networks for PoC). • Ensure NLB can reach those ports and ENIs. 4) Announced IP / ICE candidates • Configure LiveKit to advertise the NLB’s public DNS/IP so clients receive reachable ICE candidates. • Verify ICE candidates in client logs during test connections. 5) Autoscaling & metrics (practical approach) • Export a custom metric: active peer count per task or aggregated across tasks. If LiveKit exposes metrics, use a sidecar or script to push the peer count to CloudWatch. • Autoscaling policy: ◦ Scale-out: when aggregated active peers > X per task (e.g., scale to keep ~200–300 peers per task as a starting point) ◦ Scale-in: use conservative cooldown (e.g., 10–15 minutes) to avoid dropping active sessions • Start with a minimum of 0 tasks if you accept cold starts; set minimum >0 if you need lower latency for spikes. Sizing heuristic for ~1000 concurrent users • Start with a target of ~200–300 peers per 4 vCPU / 8–16 GB task as a hypothesis. • That suggests 4–5 tasks to handle 1000 peers if the per-task capacity matches the hypothesis. • Run load tests and measure CPU, memory, and packet loss to refine per-task capacity — increase tasks rather than single large tasks. Cost-conscious tips • Allow scaling to zero if cold starts are acceptable; otherwise keep a small baseline. • Use smaller tasks for better granular scaling (easier to add capacity for spikes). • Compare Fargate costs vs small EC2 spot-backed ASG for sustained baseline — Fargate is operationally simpler but can be pricier at scale. Testing checklist (short) 1. Deploy 1 task behind NLB; verify clients can connect, gather ICE candidates, and see media (test from different networks). 2. Instrument LiveKit to export peer counts to CloudWatch. 3. Run load tests, ramp to 1000 connections, monitor CPU, memory, packet loss, connection time, and ICE failures. 4. Test scaling: simulate bursts and ensure new tasks register and accept traffic via NLB. 5. Test task termination while clients are connected to observe reconnect behavior and user impact. Limitations & risks to watch • No official documentation/testing for Fargate — you may encounter UDP limitations or unexpected NAT/ENI behaviors. • Without TURN, some clients (especially behind symmetric NATs) will fail to establish media. • LiveKit session continuity may break on task termination; client reconnect/resume handling is required. If you’d like next, I can provide either: • A sample ECS task definition JSON snippet + minimal CloudFormation outline for the NLB and Fargate service (UDP+TCP), or • A focused list of exact LiveKit environment variables to set for PoC (UDP port ranges, announced IP, API keys), or • A short script example that pushes LiveKit peer counts to CloudWatch for autoscaling. Which of those three would you like next?