what is the ideal pod configuration for livekit?

Thanks for reaching out, Utkarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

currently i have the same Agents running on 2 pods with the below resources config of each pod:

resources:
        limits:
          memory: 4Gi
        requests:
          cpu: 500m
          memory: 1Gi

now i want to scale it 6x times, what should be the ideal pod configuration and how many calls i will be able to handle?

Thanks — I’m responding to your question about how to scale your LiveKit Agents 6x and what pod sizing will give you in concurrent calls. Current config (per pod) • requests:

cpu: 500m

,

memory: 1Gi

• limits:

memory: 4Gi

• You currently run 2 pods with that config. Recommended baseline for scaling Agents • Target per-pod resources: 4 CPU cores and 8 GiB RAM. ◦ This is the recommended baseline for LiveKit Agents running voice AI workloads and handles roughly 10–25 concurrent jobs per pod, depending on which features are active (enhanced noise cancellation, turn detection, recording, etc.). Capacity estimate for 6x scale • If you scale to 6 pods and use the recommended 4 CPU / 8 GiB pod: ◦ Per pod: ~10–25 concurrent calls ◦ Total for 6 pods: ~60–150 concurrent calls Notes and caveats • The 10–25 range depends on workload specifics: ◦ Light audio-only processing with minimal extra features trends toward the higher end. ◦ CPU-heavy features (enhanced noise suppression, real-time speech processing, heavy encoding) push you toward the lower end. • A real-world load test showed 30 voice agents on one 4-core / 8 GiB machine peaked around 3.8 cores and 2.8 GiB RAM — that indicates the 4/8 spec provides useful headroom for bursts. Practical next steps I recommend 1. If you want a conservative rollout, start by deploying some pods at 4CPU / 8Gi and leave others at current size; run a controlled load test to observe real CPU/RAM per call. 2. Configure Kubernetes resource

requests

and `limits`: ◦ requests:

cpu: 4

,

memory: 8Gi

◦ limits:

cpu: 4

,

memory: 8Gi

(or slightly higher if you want headroom) 1. Set up autoscaling: ◦ Horizontal Pod Autoscaler (HPA) based on CPU usage and/or custom metrics (e.g., active calls). Target a CPU utilization threshold that matches your observed per-call CPU. 1. Monitor these metrics during tests: ◦ CPU, memory, network bandwidth, packet loss, jitter, and per-call latency. 1. Share a sample session profile if you want a tighter estimate: ◦ Is it audio-only or audio+video? ◦ Which audio processing features are enabled (noise suppression, VAD, voice activity detection, transcription, etc.)? ◦ Average expected call duration and concurrency patterns (steady vs. spiky). If you share those workload details I can give a more precise per-pod concurrent-call estimate and suggested autoscaling thresholds.

one Agent server can spun how many workers? And each worker can handle how many jobs?

You’re asking how many workers one Agent server can spawn and how many jobs each worker can handle. I’ll explain how Agent servers, workers, and jobs map to processes and capacity — plus what affects how many concurrent jobs you can actually run. Key facts • An Agent server is a single Agent process (the

AgentServer

in Python or

ServerOptions

in Node). It registers with LiveKit and accepts jobs. • For each accepted job, the Agent server spawns a separate worker as a sub-process. Each worker process runs exactly one job (one agent session in one room). • There is no fixed numeric limit on how many workers (concurrent job processes) an Agent server can spawn. The practical limit is determined by the server’s hardware (CPU, memory), the agent application’s resource needs, and any configured

load_threshold

. • The LiveKit framework’s load balancing will stop routing new jobs to an Agent server once that server’s configured

load_threshold

is exceeded. What this means for you • One Agent server can run many worker processes concurrently, but each worker handles exactly one job at a time. • To know how many concurrent jobs one Agent server can support, you must profile how much CPU and RAM a single worker (job) consumes under your exact workload (features enabled, audio vs. video, processing like noise suppression or transcription). • Example baseline for voice-AI workloads: a 4 CPU / 8 GiB machine is a common recommendation and can support roughly 10–25 concurrent jobs (per machine) depending on features. Use that as a starting reference when estimating capacity per Agent server. Practical next steps I recommend 1. Run a small controlled load test with your agent code and the exact features you use (noise suppression, VAD, transcription, etc.). Measure per-worker CPU and memory. 2. Calculate concurrent jobs per server: ◦ concurrent_jobs ≈ (available_CPU_cores / avg_CPU_per_worker) and verify memory similarly. 1. Configure the Agent server’s

load_threshold

to reflect the CPU/memory utilization where you want it to stop receiving new jobs. 2. Use autoscaling (more Agent server pods) when average load approaches your threshold. If you’d like, share these details and I’ll give a more specific estimate: • Which agent features are enabled (noise cancellation, VAD, transcription, etc.)? • Are sessions audio-only or audio+video? • The CPU and memory usage you observe for a single worker during a representative job (if you have measurements). I can then help convert those numbers into an expected concurrent-job count per Agent server and suggest

load_threshold

and pod resource settings. Sources: Server options | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation | Job lifecycle | LiveKit Documentation | Server lifecycle | LiveKit Documentation

Example baseline for voice-AI workloads: a 4 CPU / 8 GiB machine is a common recommendation and can support roughly 10–25 concurrent jobs (per machine) depending on features. Use that as a starting reference when estimating capacity per Agent server • for above case does it create 10-25 workers? with each having its own thread?

Thanks — I’m responding to your question about whether the 10–25 concurrent-jobs estimate means 10–25 workers and whether each worker runs as its own thread. Short answer • Yes — the 10–25 concurrent jobs estimate refers to 10–25 worker processes (one worker per job). • Each worker runs in its own OS sub-process, not merely as a thread inside the main Agent process. So they are process-isolated, not just threads. Why that matters • Isolation: each job runs in a separate sub-process, so a crash or memory leak in one worker won’t take down other jobs or the Agent server. • Resource accounting: CPU and memory are consumed per worker process. Capacity planning should be based on per-worker CPU and RAM usage. • Scheduling/limits: Kubernetes pod-level CPU/memory limits apply to the whole Agent server process and all its child workers combined. Practical implications for the 4 CPU / 8 GiB baseline • When we say a 4 CPU / 8 GiB machine can support roughly 10–25 concurrent jobs, that implies the machine is able to run ~10–25 worker sub-processes concurrently given the typical per-job resource profile used in the load tests. • Exact counts depend on per-worker CPU and memory usage for your workload (features like noise cancellation, VAD, transcription increase CPU). Recommended next steps 1. Measure one representative job’s CPU and memory while running your agent features. 2. Estimate concurrent jobs per server: ◦ concurrent_jobs ≈ floor(available_CPU_cores / avg_CPU_per_worker) ◦ Confirm memory: concurrent_jobs × avg_memory_per_worker ≤ available_memory (leave headroom). 1. Set the Agent server

load_threshold

to the CPU/memory point where you want the server to stop accepting new jobs. 2. Run a controlled load test to validate the estimate and observe behavior (CPU, memory, and process count). If you’d like, share a single-job CPU/memory sample (or the features you enable), and I’ll estimate how many worker processes a 4 CPU / 8 GiB pod would likely support and suggest sensible

load_threshold

and pod resource settings.

So how many tasks in 1 cpu and 2 gb

Thanks — I’m responding to your question about how many concurrent tasks (jobs/workers) one Agent pod can run when it has 1 CPU and 2 GiB RAM. Short answer • Expect roughly 2–6 concurrent jobs on a pod sized 1 CPU / 2 GiB, depending on how CPU- and memory‑heavy each job is. Why that range • The 4 CPU / 8 GiB baseline commonly supports about 10–25 jobs. Scaling that baseline linearly gives a rough proportional expectation: 1 CPU / 2 GiB is about one quarter of 4 CPU / 8 GiB, so the same 10–25 range scales to ~2.5–6 jobs — rounded to 2–6 to account for real-world variance. • Actual capacity moves toward the lower end when you enable heavier processing (enhanced noise cancellation, real‑time transcription, turn detection, audio+video, etc.), and toward the higher end for light audio-only jobs with minimal processing. Important caveats and what to check • This is an estimate. The true limit depends on: ◦ Per-job CPU usage (most critical) and per-job memory. ◦ Whether jobs are audio-only or audio+video. ◦ Which features are enabled (noise suppression, VAD, ASR, recording). ◦ Network bandwidth and I/O patterns. • Each job runs as its own OS sub-process, so you’ll see one process per job and resource consumption is additive inside the pod. Recommended next steps to get a precise number 1. Run a small profiling test with one representative job and measure: ◦ Average CPU usage per job (e.g., in millicores) ◦ Average memory per job 1. Calculate capacity: ◦ concurrent_jobs ≈ floor(available_CPU_cores / avg_CPU_per_job) ◦ Verify memory: concurrent_jobs × avg_memory_per_job ≤ available_memory (leave headroom) 1. Set the Agent server

load_threshold

to the CPU or memory point where you want it to stop receiving new jobs. 2. Do a controlled ramp test (start 1 job, then add more) and watch CPU, memory, and stability. If you share: • whether these are audio-only or audio+video jobs, and • which processing features you enable (noise suppression, VAD, transcription, etc.), I can help convert that into a tighter estimate and suggest `requests`/`limits` and

load_threshold

values.

can we spawn workers as per need and reuse them?

Thanks — I’m replying about whether workers can be spawned on demand and reused for multiple jobs. Short answer • LiveKit Agents spawn a new sub-process per job by default, and that worker process handles exactly one job. Workers are process‑isolated and not reused across jobs in the default model. • Reusing workers (a worker pool) is possible only if you implement it yourself in your agent application, but it has trade-offs you should consider. Details and trade-offs • Default behavior: ◦ Each accepted job spawns a new OS sub-process (worker) and that process runs one job to completion. This provides strong isolation: a crash or memory leak in one worker won’t affect others or the Agent server. • Implementing worker reuse (pooling): ◦ You could design your agent to maintain a pool of pre-started worker processes or threads and dispatch jobs to idle workers. ◦ Benefits: lower latency for job startup (no process spawn per job) and potentially better CPU/memory amortization if workers are expensive to initialize. ◦ Drawbacks: ▪︎ Reduced isolation — a faulty job could affect the pooled worker and therefore future jobs that reuse it. ▪︎ More complex lifecycle and cleanup logic (you must safely reset worker state between jobs). ▪︎ Harder failure containment — you’ll need additional monitoring and restart logic for pooled workers. ▪︎ Resource accounting and the Agent framework’s load balancing expect per-job process isolation; you’ll need to ensure your

load_threshold

and health checks reflect pooled-worker behavior. • Alternatives to pooling: ◦ Keep the default per-job process model and rely on autoscaling and the recommended sizing (e.g., 4 CPU / 8 GiB per machine ≈ 10–25 concurrent jobs) to meet capacity needs. ◦ If startup latency is the main concern, consider a hybrid: keep short-lived per-job processes for isolation but warm a small number of workers for particularly latency-sensitive tasks. ◦ Optimize the worker initialization path (lazy-load heavy libraries or cache state) to reduce per-job startup time without changing the process model. Practical recommendations 1. If you need strong fault isolation and simpler failure handling, stick with the default per-job sub-process model. 2. If you need lower startup latency and accept added complexity, implement a worker pool with strict reset and monitoring logic. Make sure to: ◦ Reset worker state between jobs. ◦ Restart workers after a set number of jobs or on any error. ◦ Expose health checks and update

load_threshold

behavior accordingly. 1. Whatever approach you pick, run a controlled test (ramp-up) and measure CPU, memory, job latency, and failure modes to validate behavior. If you want, tell me: • whether startup latency or resource efficiency is your primary concern, • which agent features are enabled (noise suppression, ASR, etc.), and • whether you prefer isolation over lower latency. I can then suggest a concrete pooling design or tuning strategy and sample settings (when to restart pooled workers, pool size relative to CPU/memory, and how to set

load_threshold

).