What's the expected consumption (CPU and RAM) per job for a LiveKit worker using TTS, STT, LLM, VAD and TurnDetection Model. Currently we're self hosted on GKE (Google Kubernetes Engine) node are n2-standard-4 machines (4 vCPU and 16GB RAM) and our consumption are far from what it's published in LiveKit docs.

Thanks for reaching out, Tarek-Asis. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

1. We use the following models: a. STT: Deepgram Nova-3 b. LLM: Azure GPT-4o c. TTS: Cartesia Sonic-3 d. VAD: ONNX 2. Models run in-process in the same pod 3. ~10 jobs per worker and average audio length per turn ~4 seconds 4. Only the model used by ONNX and TurnModelDetection 5. ~10 concurrent calls causes ~66% CPU consumption and 46% RAM consumption

Thanks — this is helpful. I’ll respond to your points and give actionable next steps. Summary of your setup I’m addressing: • Models: Deepgram Nova-3 (STT), Azure GPT‑4o (LLM), Cartesia Sonic‑3 (TTS), ONNX VAD, TurnDetection (ONNX). • All models run in‑process in the same pod. • ~10 jobs/streams per worker, ~4s average turn. • Only ONNX/TurnDetection use local acceleration. • Observed at ~10 concurrent calls: ~66% CPU, ~46% RAM on n2‑standard‑4 (4 vCPU / 16 GB). What this implies (short): • The TurnDetector needing <500 MB RAM fits with your RAM headroom; the rest of the memory is likely used by the in‑process runtime, language model clients, TTS buffers, and system overhead. • CPU at ~66% for 10 concurrent short turns suggests inference or audio processing is CPU‑bound (multiple small synchronous tasks add up), not a single large model hogging CPU. Practical per‑job estimate (based on your measurements and concurrency): • Rough observed per‑job average (approximate): ~0.25 vCPU and ~0.7–1.0 GB RAM per concurrent job. ◦ Rationale: 4 vCPU * 0.66 ≈ 2.64 vCPU used total → ~0.264 vCPU/job. ◦ 16 GB * 0.46 ≈ 7.4 GB used total → subtract ~2–3 GB baseline for OS/process → ~4.4–5.4 GB for jobs → ~0.44–0.54 GB/job; conservatively allow ~0.7–1.0 GB/job for buffers/peaks. • These are conservative approximations you can use for right‑sizing and requests/limits. Recommendations (next actions you can take) 1. Split heavy components into separate pods: ◦ Move TTS, STT/LLM clients, and ONNX inference into dedicated pods or sidecars so each can scale independently. This reduces in‑process contention and makes resource allocation predictable. 1. Use pod resource requests/limits: ◦ Start with requests = 0.3 vCPU and 0.8 GB RAM per concurrent job and limit = 0.6–1.0 vCPU and 1.5–2.0 GB RAM. For 10 concurrent jobs on one worker, request ~3 vCPU and ~8 GB memory. 1. Increase concurrency granularity: ◦ If each worker must serve 10 concurrent short turns, consider scaling to more workers with fewer concurrent jobs per worker (e.g., 4–6 jobs/worker) to reduce CPU peaks. 1. Measure per‑process breakdown: ◦ Capture cgroup or container metrics to see CPU and RSS per thread/process (e.g.,

kubectl top pod

,

ps aux --sort=-%mem

inside pod, or cAdvisor metrics). This will show whether TTS/STT client libraries or audio I/O are the main CPU consumers. 1. Profile short‑turn overhead: ◦ Short 4s turns create higher per‑turn overhead (setup/teardown, encoding). Batch or reuse model sessions if possible to reduce per‑turn cost. 1. Consider CPU optimized nodes: ◦ Use compute‑optimized instance types (higher vCPU/core perf) rather than general purpose if inference is CPU bound. 1. Use model acceleration / smaller variants where possible: ◦ If latency tolerant, offload LLM calls to remote Azure (they already are) but ensure client I/O and encoding are asynchronous so they don’t block CPU. For TTS/STT, choose lighter local models or quantized/ONNX runtimes if available. If you want, I can: • Help compute specific pod requests/limits for a chosen target concurrency per node (give me target jobs/node). • Provide a short checklist of exact commands/metrics to collect for a precise per‑job profiling run. Which option would you like next?

Before jumping to any other topic, I'd like to remark that I'm using as baseline the benchmark published in LiveKit docs: • 30 agents each placed in their own LiveKit Cloud room. • 30 simulated user participants, one in each room. • Each simulated participant published looping speech audio to the agents. • Each agent subscribed to the incoming audio of the user and ran the Silero VAD plugin. • Each agent published their own audio (simple looping sine wave). • One additional user participant with a corresponding voice AI agent to ensure subjective quality of service. This test ran all agents on a single 4-Core, 8GB machine. This machine reached peak usage of: • CPU: ~3.8 cores utilized • Memory: ~2.8GB used It's a consumption of 0.12CPU/job and 0.09GB/job. I know I'm using also the Turn Detection Model but I expected to have similar results and that's not the case. Our CPU consumption is x2 higher and our RAM consumption is ~x5 higer. Is it expected or should I try to optimize our code?

Thanks — I’ve reviewed your baseline and your detailed setup. Quick answer: your higher CPU (~2x) and RAM (~5x) usage is expected given the extra components you’re running, not necessarily a sign of buggy code. Why this is expected (brief): • The LiveKit benchmark you cited used only Silero VAD and a lightweight audio loop. That baseline (0.12 CPU / 0.09 GB per job) does not include Turn Detection, STT, TTS, or an LLM. • The Turn Detection model is a transformer (~135M params) that runs on CPU and adds substantial memory (~≈400 MB) and CPU overhead per agent. • Running STT (Deepgram), TTS (Cartesia), and LLM calls alongside in‑process increases both memory and CPU: even if LLM inference happens remotely, local client libraries, audio encoding/decoding, buffers, and per‑turn setup add cost. • Short 4s turns increase per‑turn overhead (setup/teardown, encoding), which raises CPU usage vs long continuous streams. Given your reported metrics (10 concurrent jobs → ~66% CPU, ~46% RAM on n2-standard-4), your observed per-job usage (~0.24 vCPU and ~0.4–0.7 GB RAM actual; conservatively ~0.25 vCPU and ~0.7–1.0 GB accounting for peaks) aligns with expectations for this stack. Recommended next steps (concise): • Ensure models are prewarmed and shared across jobs (don’t reinstantiate heavy models per agent). • Move heavy components into separate pods/sidecars where possible (TTS, STT clients, ONNX inference) so they can scale independently. • Use compute‑optimized nodes rather than burstable instances to avoid CPU throttling. • Set pod requests/limits. Start with requests ≈ 0.3 vCPU / 0.8 GB per concurrent job and limits ≈ 0.6–1.0 vCPU / 1.5–2.0 GB; for 10 jobs target requests ≈ 3 vCPU / 8 GB. • Profile per-process/container usage (cgroup,

kubectl top pod

, in‑pod `ps`/`psutil`) to confirm which component uses the most CPU/RAM. • Reduce per‑turn overhead: reuse sessions, batch where possible, or reduce jobs-per-worker. If you’d like, I can: • Calculate exact requests/limits for a target jobs-per-node number you choose. • Provide the exact commands/metrics to collect for a per-component profile. Which would you prefer next? Sources: Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Improving voice AI&amp;#39;s turn detection with transformers | Improved End-of-Turn Model Cuts Voice AI Interruptions 39% | LiveKit turn detector plugin | LiveKit docs | Silero VAD plugin | LiveKit docs

Is there any example of running this heavy components in different pods? As far as I know it's part of the agent and cannot be run in a different machine/process

Thanks — I’m responding to your question about whether heavy components can be run in different pods/processes instead of in‑process in the agent. Short answer: • LiveKit Agents are designed to run STT, TTS, LLM, VAD, and TurnDetection inside the same agent process. There are no documented examples or supported patterns to split those components across separate pods as first‑class services. Why: • The agent pipeline expects low‑latency access to streaming audio, VAD/STT output, and turn detector inputs; these components are tightly integrated into the AgentSession flow, so splitting them requires custom networking, serialization, and orchestration that the framework does not provide out of the box. What you can do instead (practical options): • Prewarm and share models inside each worker process so heavy models are loaded once per process (not per session). • Use horizontal scaling: run more agent pods with fewer concurrent sessions per pod. • Move only truly external workloads off‑process: STT/LLM/TTS calls are already external when you use cloud providers (Deepgram, Azure, Cartesia). Ensure those calls are async so they don’t block local CPU. • Choose lighter or quantized model variants for local components (VAD/TurnDetector) or reduce model sizes where acceptable. • If you still need a split architecture, you can build a custom RPC/HTTP wrapper around a local model (e.g., VAD/TurnDetector) and call it from the agent, but this is a custom engineering effort (not a supported LiveKit pattern) and will require handling streaming, batching, and latency yourself. If you want, I can: • Show a short example pattern for a custom wrapper service (HTTP/RPC) for VAD or TurnDetection and the tradeoffs to watch for, or • Compute recommended pod sizing and concurrency targets if you move to more horizontal workers. Which would you prefer? Sources: Agents (Python) | livekit/livekit_composite | DeepWiki | Introducing LiveKit Inference: A unified model interface for voice AI | Agents Framework | livekit/livekit_composite | DeepWiki | examples/frontdesk | livekit/livekit_composite | DeepWiki | examples/drive-thru | livekit/livekit_composite | DeepWiki