https://livekit.io logo
Join Slack
Powered by
We are using livekit Scale plan and now on boardin...
# ask-ai
b
We are using livekit Scale plan and now on boarding Enterprise as well. Jesse and Jason is working with us. But, we could find this issues. The major issue is; When we run the system with around 15 concurrent calls, it scales up to 4 replicas based on both 50% CPU and 50% memory utilization. My concern here is that as concurrency increases, more replicas will be required. If LiveKit (or our service) takes time to spin up new containers, this could negatively impact user experience during traffic spikes. I’d love to hear your thoughts on: • How the race condition could affect calls where participants are still active • Whether there are any obvious CPU-heavy paths in the current implementation • What strategies you’d recommend to mitigate scaling and startup latency issues under higher concurrency Looking forward to your guidance. my start up script: Run the LiveKit agent
Copy code
```
```cli.run_app(

    WorkerOptions(

        *entrypoint_fnc***=**entrypoint,

        *prewarm_fnc***=**prewarm,

        *worker_type***=**WorkerType.ROOM,

        *initialize_process_timeout***=**120.0,

        *agent_name***=**settings.agent_name,

    )
t
Thanks for reaching out, Yethu. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great, specific set of concerns. I’ll respond to each of your three points and then give focused, actionable recommendations you can apply right away. I’m responding to your questions about race conditions during scale-up, CPU-heavy paths in your deployment, and strategies to reduce startup/scale latency. Race conditions and active calls during scale-up • Active calls are not moved or interrupted when new replicas spin up. New jobs are routed only to available workers; running rooms remain on their original worker until they finish. • Your 
initialize_process_timeout=120.0
gives the process up to 120s to finish initialization/prewarm which reduces the chance of a dispatch failing because the worker wasn’t ready. • If a job dispatch does fail to connect (for example the worker doesn’t establish the room connection within the dispatch window), the job will be retried on another worker. That provides fault tolerance but it’s a path that can add latency for that particular session if it occurs. CPU-heavy paths to watch for • Real-time inference and audio/video processing are the biggest CPU consumers: ◦ VAD (Voice Activity Detection) or any model loaded per-request is CPU-heavy unless prewarmed and reused. ◦ STT/TTS/LLM inference pipelines and any synchronous model calls are high CPU users. ◦ Video processing (transcoding, background effects) is also expensive if in use. • Burstable instance types (e.g., AWS T3/T4) can appear to have CPU headroom but will throttle under sustained load; that causes timeouts and degraded latency even when observed CPU% looks moderate. Strategies to mitigate scaling and startup latency 1. Keep warm workers (prevent cold starts) ◦ Maintain a minimum number of prewarmed workers so new sessions can be placed immediately during spikes. This avoids container spin-up latency entirely for those requests. ◦ You already use a 
prewarm_fnc
— ensure heavy model loading (VAD, STT clients, etc.) happens inside that prewarm function and is stored on the process object (e.g., 
proc.userdata
) for reuse. 1. Optimize what is loaded on startup ◦ Load large models or libraries during prewarm, not at first request. ◦ Reuse loaded models across sessions from the process-level cache. 1. Use predictable instance types ◦ Prefer non-burstable CPU instances (e.g., c5/m5 families on AWS) for agent workers to avoid CPU credit exhaustion and throttling. 1. Tune autoscaling & capacity ◦ Keep a small buffer of idle warm replicas beyond your steady-state concurrency so the autoscaler has time to spin more up for sudden spikes. ◦ If you expect predictable spikes, pre-scale ahead of the window (schedule capacity increase) so new capacity is already warm. 1. Limit per-process concurrency for heavy inference ◦ If a worker hosts multiple heavy inference sessions concurrently, that can cause contention. Consider limiting concurrency per worker for model-heavy workloads or isolating inference onto dedicated workers. 1. Graceful draining and deployments ◦ Ensure workers are drained (no new jobs assigned) before termination so active calls finish on their original worker. ◦ Use blue/green or phased rollouts to test new builds without impacting active sessions. 1. Monitor useful signals (not just CPU%): ◦ Session count per worker, inference queue length, model latency, and process start time. ◦ For AWS, monitor CPU credit balance if you use burstable instances. Quick checks and next steps you can run now 1. Verify your prewarm actually loads and caches heavy components (VAD/STT/LLM) into 
proc.userdata
. 2. Confirm your cloud VM types are non-burstable for production-critical agents. 3. Keep a baseline of prewarmed workers (a small pool) to absorb short spikes. 4. Instrument per-worker metrics for: ◦ active room count ◦ inference requests/sec and latency ◦ process startup time (from spawn to "ready") 1. If you see dispatch retry events, capture logs of the dispatch window and worker readiness—those will pinpoint whether the issue is startup slowness or transient network/connect failures. What I can help with next • Review your 
prewarm_fnc
code to confirm models are being loaded once and reused. If you paste it, I’ll point out any gaps. • Suggest concrete autoscaling buffer sizes based on your observed 15 concurrent calls baseline (I’ll need current per-room CPU/memory numbers and the average model usage per room). • Help interpret logs if you can share a snippet of a dispatch retry or a worker startup timeline. If you’d like, paste your actual 
prewarm_fnc
and the exact worker VM type you’re using (e.g., 
t3.medium
, 
c5.large
), and I’ll give specific, actionable adjustments. Sources: Deploy and scale agents on LiveKit Cloud | Agents Framework | livekit/livekit_composite | DeepWiki | Server options | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS
b
what is the livekit recommended workeroptions for production
t
Thanks — I’ll respond to your question about recommended 
WorkerOptions
for production and give concise, practical defaults you can use right away. Before I give a final sample, quick question: are you running agents in Python or Node/TypeScript? I’ll include both, but let me know if you want only one. Key production principles • Prewarm heavy resources (models, SDK clients) so job startup is fast. • Use the right worker type for your workload (room-level vs per-publisher). • Keep graceful drain time long enough to let sessions finish. • Limit per-worker concurrency for heavy inference workloads or isolate inference onto dedicated workers. • Use non-burstable instance types in production for predictable CPU. • Store secrets via environment variables, not in code. Recommended WorkerOptions — production defaults 1) Python (concise example)
Copy code
from livekit.agents import WorkerOptions, WorkerType, cli

cli.run_app(WorkerOptions(
    entrypoint_fnc=entrypoint,          # your room entry function
    prewarm_fnc=prewarm,                # load models / clients here
    worker_type=WorkerType.ROOM,        # ROOM for one agent per room
    initialize_process_timeout=120.0,   # allow up to 120s to initialize / prewarm
    drain_timeout=1800.0,               # 30 minutes to let sessions finish gracefully
    # other runtime options you set via env or platform (e.g., agent_name)
))
2) Node / TypeScript (concise example)
Copy code
import { WorkerOptions, WorkerType, cli } from '@livekit/agents';

cli.runApp(new WorkerOptions({
  entrypoint_fnc: entrypoint,          // your entry handler
  prewarm_fnc: prewarm,                // load heavy models/clients here
  worker_type: WorkerType.ROOM,        // ROOM or PUBLISHER as needed
  initialize_process_timeout: 120.0,   // seconds
  drain_timeout: 1800.0,               // seconds (30 min)
}));
What to put in 
prewarm_fnc
• Load VAD, STT, TTS, or LLM clients here and save to process-level storage so they’re reused: ◦ Python: 
proc.userdata["vad"] = silero.VAD.load()
◦ Node: 
proc.userData.vad = await silero.VAD.load()
• Keep network calls/long downloads in prewarm so per-job latency is minimal. WorkerType guidance • WorkerType.ROOM — use when one worker should own a full room (recommended for most agent-driven rooms). • WorkerType.PUBLISHER — use if you need per-participant isolation (e.g., per-user heavy processing). Timeouts and graceful shutdown • 
initialize_process_timeout = 120s
gives ample time for prewarm. Increase if your models take longer to fetch. • 
drain_timeout = 1800s
(30 minutes) avoids cutting off active sessions during deployments. Reduce only if your sessions are always short. Scaling and concurrency controls • Keep a small pool of prewarmed workers to absorb spikes (cold-start prevention). • Limit concurrent heavy inference sessions per worker; move heavy inference to dedicated workers if needed. • Monitor: active rooms per worker, inference latency, process startup time — scale thresholds should use these metrics in addition to CPU/memory. Deployment & infra tips • Non-burstable VM types (c5/m5 families) for predictable CPU. • Multi-stage Docker builds that pre-download models reduce runtime initialization. • Run multiple workers (same agent_name) for redundancy; use environment variables for secrets. If you want, I can: • Return a fully worked Docker + WorkerOptions example that pre-downloads models and sets environment variables. • Propose specific per-worker concurrency limits and the number of prewarmed workers if you share: average CPU/RAM per room, and whether you use VAD/STT/LLM per room. • Provide a review of your current 
prewarm_fnc
(paste it) and recommend exact changes. Sources: Silero VAD plugin | LiveKit Documentation | Server options | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki | Self-hosted deployments | LiveKit Documentation
b
what about the job process type
t
Thanks — I’m responding to your follow-up about the “job process type.” I’ll explain what a job process is, how it behaves, how it relates to WorkerOptions, and which production practices affect job processes so you can optimize startup, isolation, and scaling. What the job process is • A job process is the isolated, short-lived subprocess spawned by the agent server to run a single job (for example, a room session). It executes your 
entrypoint
logic and terminates when the session ends. • This isolation ensures failures or memory leaks in one job don’t crash other jobs running on the same agent server. Job process lifecycle and behavior • Start: spawned when the agent server accepts a dispatch for a job. • Initialization: it runs any job-level startup steps and becomes “ready” to handle the session. • Run: it executes your 
entrypoint
(room/session handling) until participants leave or the session ends. • Termination: the process exits when the job completes or if it’s forcibly terminated (e.g., worker shutdown after drain timeout). How WorkerOptions affects job processes • Worker/server-level settings (via 
WorkerOptions
or agent config) indirectly control job processes: ◦ 
prewarm_fnc
/ 
prewarm
runs in the long-lived agent server process to preload models/clients so job processes can start faster. ◦ 
initialize_process_timeout
controls how long a process is allowed to initialize. Increase this if your job process needs more time to start (e.g., large models). ◦ 
drain_timeout
controls how long the server will wait for in-flight job processes to finish during shutdown/deploys. ◦ 
worker_type
(ROOM vs PUBLISHER) determines dispatch granularity and therefore the number and lifetime of job processes: ▪︎ `WorkerType.ROOM`: one job process per room (recommended for most agent scenarios). ▪︎ `WorkerType.PUBLISHER`: one job process per publisher/participant (useful for per-user isolation). Practical implications for production • Isolation and fault tolerance: because each job is a separate process, resource exhaustion or crashes are contained to that job. This improves reliability but means you must still monitor per-job memory/CPU usage. • Startup latency: job processes add startup time on dispatch. Mitigate this by prewarming heavy dependencies at the server level and keeping a pool of prewarmed workers to reduce cold starts. Also tune 
initialize_process_timeout
to allow adequate startup time. • Scaling behavior: more concurrent jobs → more job processes → more replicas may be required. Use non-burstable instances and reserve prewarmed capacity to avoid slow scale-ups. • Concurrency and resource limits: if each job spawns heavy inference or video processing, consider limiting the number of concurrent jobs per worker, or isolate heavy workloads onto dedicated agent workers so one process’s heavy CPU usage doesn’t impact others. Recommended production practices related to job processes • Prewarm heavy resources in 
prewarm_fnc
and store them at the agent/server level so job processes start faster (avoid loading models per job). • Choose 
WorkerType
appropriately: ◦ Use 
ROOM
for room-level agents (fewer, longer-lived job processes). ◦ Use 
PUBLISHER
only if you need per-publisher isolation (more, shorter-lived processes). • Tune timeouts: ◦ `initialize_process_timeout`: set enough time for model loading during process init (e.g., 60–120s or higher if needed). ◦ `drain_timeout`: keep long enough (default 30 min) to allow sessions to finish during deploys. • Use process-level monitoring: ◦ Track per-process startup time, memory and CPU, and job counts to set autoscaling thresholds. • Deploy multiple agent instances with the same 
agent_name
for redundancy. Or dedicate certain workers for heavy-inference jobs to avoid noisy-neighbor effects. If you want, I can: • Review your current prewarm/entrypoint code to spot opportunities to move heavy initialization out of the job process. • Recommend concrete initialize_process_timeout and concurrency limits based on your per-room CPU/memory numbers — paste a sample of your per-room resource usage and I’ll propose numbers. • Suggest a deployment pattern (number of prewarmed workers vs. autoscale buffer) tailored to your observed 15 concurrent sessions baseline. Which of the three would you like me to start with? Sources: Job lifecycle | LiveKit Documentation | Server options | LiveKit Documentation | Agents Framework | livekit/livekit_composite | DeepWiki
b
_job_executor_type thread or process?_
t
Short answer: job executors run as separate processes, not threads. Why that matters • Each job (for example, a room session) is spawned into its own process for strong isolation. A crash, memory leak, or heavy CPU use in one job won’t take down other jobs on the same agent server. • Process isolation increases reliability and makes it safe to run untrusted or heavy workloads (inference, video processing) without risking the host agent. How this affects your production setup • Startup latency: spawning a process adds startup time. Mitigate with 
prewarm
(load heavy models in the long‑lived server) and keep a pool of prewarmed workers to avoid cold starts. • Resource accounting: with processes you’ll see one OS process per job — plan CPU/RAM per-process and autoscale accordingly. • Fault handling: a failing job process can be restarted or retried without affecting other jobs. • WorkerOptions interaction: 
prewarm_fnc
runs in the long-lived agent process to reduce per-job initialization. Tune 
initialize_process_timeout
and 
drain_timeout
to match your model load times and session durations. Practical recommendations • Use 
WorkerType.ROOM
for one process per room (fewer, longer-lived processes) unless you need per-publisher isolation. • Set 
initialize_process_timeout
high enough for your job process to initialize (60–120s is common if you load models). • Keep a small set of prewarmed workers to absorb spikes and avoid cold process starts. • Prefer non-burstable instance types (c5/m5 families) for predictable CPU behavior. If you want, I can: • Review your 
prewarm_fnc
to confirm heavy loads are done in the server process (paste it here). • Suggest concrete timeouts and prewarm pool sizes based on your per-room CPU/RAM numbers. Sources: Job lifecycle | LiveKit Documentation | An open source stack for real-time multimodal AI | Server lifecycle | LiveKit Documentation | Modifying LLM output before TTS | LiveKit Documentation | Server options | LiveKit Documentation
b
WARNINGlivekit.agentsprocess memory usage is high {"message": "process memory usage is high", "level": "WARNING", "name": "livekit.agents", "memory_usage_mb": 530.46875, "memory_warn_mb": 500, "memory_limit_mb": 0, "pid": 1953, "job_id": "AJ_qkzKXJRHWybB", "room_id": "RM_apyivUxySUAU", "timestamp": "2026-02-13T072511.596934+00:00"} INFOcore.agent sessionUser listening INFOcore.agent sessionUser listening
t
Thanks — I’m responding to the memory warning you pasted: 
WARNING:livekit.agents:process memory usage is high
with memory_usage_mb ~530.5 vs memory_warn_mb 500. I’ll explain what that means, why it happens, and concrete steps you can take right now. What the warning means • The LiveKit Agents runtime detected a job process exceeded the 500 MB warning threshold and emitted an informational warning. The process is not automatically killed (memory_limit_mb = 0), but this is an early signal that memory pressure is rising for that job. • Each job runs in its own OS process, so this is isolated to the specific room/job (room_id RM_apyivUxySUAU, job_id AJ_qkzKXJRHWybB). Other jobs remain unaffected unless overall host memory is exhausted. Common causes • Loading large models (VAD, STT, TTS, LLM) into memory per job. • Loading models inside the job/entrypoint instead of in the server prewarm step, causing repeated memory allocations. • Long-running sessions that accumulate buffers, transcripts, or context objects. • Video processing or other media pipelines that allocate significant RAM. • Using small/burstable VMs where memory and CPU become constrained. Immediate actions to take 1. Confirm where models are loaded ◦ Ensure heavy models are loaded in your `prewarm_fnc`/`setup_fnc` (server-level) and stored in 
proc.userdata
/ 
proc.userData
. This avoids reloading per job and reduces per-process initialization overhead. 1. Inspect session memory growth ◦ Check whether memory grows steadily over time (possible leak) or spikes at startup (model load). Capture memory samples for the job process over the session lifetime. 1. Increase instance memory or change instance type ◦ Move from burstable instances (t3/t4) to non-burstable, compute-optimized VMs (e.g., c5/m5 families) and add RAM if you frequently hit the 500 MB warning. 1. Add a small pool of prewarmed workers ◦ Keep several warm/idle worker instances so new jobs are placed on already-initialized processes rather than spawning new processes under load. 1. Limit per-worker concurrency for heavy inference ◦ If a worker handles multiple heavy jobs, isolate inference-heavy jobs on dedicated workers or lower the number of concurrent jobs per worker. 1. Tune timeouts and monitoring ◦ Ensure 
initialize_process_timeout
allows enough time for model prewarm (60–120s or higher if needed). ◦ Track metrics per process: startup time, peak memory, RSS vs virtual memory, and number of active rooms. If the warning persists or memory steadily increases during a session • Look for unbounded data structures in your agent logic (e.g., storing full audio buffers, never-cleared transcript arrays, or accumulating context). • Ensure streams and file handles are closed on session end. • If you load multiple model instances per job (e.g., creating a new STT client per request), reuse a single client stored in 
proc.userdata
. Recommended monitoring / Kubernetes practices • Set sensible requests/limits in your pod spec (example): resources: requests: memory: "1Gi" cpu: "500m" limits: memory: "2Gi" cpu: "2000m" • Scale based on per-process RSS and job counts, not CPU% alone. • Add alerting on sustained per-process RSS above your expected baseline (e.g., alert if >75% of requested memory for >5 minutes). Example: move model load into prewarm (Python)
Copy code
def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()    # load once, share
    proc.userdata["stt"] = deepgram.STT()       # reuse client
    # ... other heavy initializations

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    vad = ctx.proc.userdata["vad"]
    stt = ctx.proc.userdata["stt"]
    session = AgentSession(vad=vad, stt=stt, ...)
    await session.start(ctx.room)
What I can help with next • Review your 
prewarm_fnc
and 
entrypoint
(paste them) and point out where to move allocations or close resources. • If you can share memory samples over time for a job (startup vs 5/15/30 minutes), I’ll help determine if it’s a startup spike or a leak and recommend exact instance sizes and prewarm pool counts. If you’d like, paste your 
prewarm_fnc
and a short snippet of the entrypoint or how you instantiate STT/VAD/LLM clients and I’ll review for memory optimizations. Sources: examples/drive-thru | livekit/livekit_composite | DeepWiki | Job lifecycle | LiveKit Documentation | OpenAI and LiveKit partner to turn Advanced Voice into an API | Self-hosted deployments | LiveKit Documentation | LiveKit turn detector plugin | LiveKit Documentation
4 Views
Open in Slack
PreviousNext
Powered by