https://livekit.io logo
Join Slack
Powered by
WARNING:livekit.agents:process memory usage is hig...
# ask-ai
f
WARNINGlivekit.agentsprocess memory usage is high
t
Thanks for reaching out, Vishalli. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I see the warning you’re seeing: 
WARNING:livekit.agents:process memory usage is high
. I can help investigate and give practical next steps. To narrow down whether this is expected or a problem, I have a few questions and some immediate checks you can run. Quick checklist you can run now • Check total system memory and current usage (e.g., 
free -h
or your cloud console metrics). • Identify the agent process memory: 
ps aux --sort=-%mem | head
or use your process manager to see resident set size (RSS) for 
livekit.agents
. • Note whether memory climbs steadily over time or spikes during certain events (call starts, plugin usage, peak concurrent sessions). Questions to help diagnose • Is this happening in local development or in a self‑hosted/production cloud environment? • How many agent processes or agent jobs are running on this host? • Which agent features/plugins are enabled (for example: VAD, noise cancellation, the LiveKit turn detector)? • How much RAM does the host have and is it a burstable or dedicated instance type? • When does the warning appear — under load spikes, after long uptime, or immediately after start? • Any other symptoms (slow media, dropped sessions, OOM kills in dmesg/syslog)? What’s likely and what to expect • Agents using features like noise cancellation or turn-detection can use noticeably more CPU and memory. Agents with those enabled require more resources than a minimal agent. • In a load test, 30 agents on a 4‑core, 8GB machine used ~2.8 GB at peak — so memory per-agent can vary with workload and enabled features. • The turn detector alone typically uses under 500 MB when run locally in a custom deployment. Short-term mitigations • If memory is temporarily high but not exceeding system capacity, you can monitor and let the process continue while collecting metrics. • If memory usage approaches or exceeds available RAM: reduce concurrent agents on that host, disable heavy plugins temporarily, or move agents to larger instances. • Ensure graceful draining is configured for agent shutdowns (there’s a drain timeout option to avoid abrupt session termination). If you’d like, I can suggest specific configuration or sizing recommendations based on your answers to the questions above (number of agents, enabled plugins, host RAM). Share those details and I’ll provide targeted next steps. Sources: Self-hosted deployments | LiveKit Documentation | LiveKit turn detector plugin | LiveKit Documentation | Server options | LiveKit Documentation
f
it is happening on my prod
t
Thanks — I hear you: this is happening in production and you need actionable help. I can help narrow down root causes and give targeted steps, but I need a few details to recommend the best next actions. Please share the following about your production deployment: • Is this a single host or a cluster? Cloud provider and instance type (e.g., AWS m5.large, GCP n2-standard-4, etc.). • Total RAM on the host and whether instances are burstable (t3/t4) or fixed. • How many agent processes or agent jobs run on that host at peak? • Which agent features/plugins are enabled (for example: noise cancellation, VAD, LiveKit turn detector, any local ML models)? • When the warning appears: immediately on startup, after long uptime, or during traffic spikes (call starts / concurrent joins)? • Any recent changes (config, deployment, new plugin, code changes) before this started happening. • Output of a quick check you can run now: ◦ 
free -h
(system memory) ◦ 
ps aux --sort=-%mem | head -n 10
(top memory consumers) ◦ Check system logs for OOM kills: 
dmesg | grep -i kill
or 
journalctl -k | grep -i oom
Immediate guidance while you gather details • If heavy features are enabled (noise cancellation, large local models, turn detector), expect higher memory per-agent; consider disabling or moving those to larger instances temporarily. • If memory grows slowly over time, that could indicate a memory leak; collect memory usage graphs (host metrics) and agent logs over time. • If memory spikes under load, scale out (more agents/hosts) or scale up instance size; avoid running too many agents on an 8GB machine if each agent is resource heavy. • If you’re on burstable instances and see degraded performance, switch to dedicated instance types to avoid CPU throttling that can affect process behavior. If you share the details and the command outputs above, I’ll: • Analyze the numbers and point out whether usage is expected or abnormal. • Recommend a short-term mitigation (instance sizing, plugin disablement, or drain/rotation). • Suggest long-term actions (autoscaling/observability tuning, prewarming models, or configuration changes). You’re on the right track — share those details and I’ll give a concrete plan to stabilize production. Sources: Self-hosted deployments | LiveKit Documentation | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Streamline troubleshooting with Agent Observability | TTS Metrics | LiveKit Documentation
f
free -h
there is still 9g memory is free
t
Thanks — good to know there’s still about 9 GB free. That helps narrow things down. Quick takeaway • If the host still has 9 GB free, the system overall is not out of memory. The warning 
WARNING:livekit.agents:process memory usage is high
is likely about a specific agent process using more memory than expected, not the host running out of RAM. To diagnose the agent-level issue, could you run and share the following outputs? 1. 
ps aux --sort=-%mem | head -n 10
— top memory-consuming processes (shows which process is using the memory). 2. 
top -b -n 1 | head -n 20
— quick snapshot of CPU/memory usage. 3. 
pmap <PID> | tail -n 20
(replace <PID> with the agent process id from step 1) — memory map for the agent process. 4. 
journalctl -u <agent-service> --since "1 hour ago" | grep -i memory
(or relevant agent logs) — any memory-related log lines around the warning. Also answer these context questions: • How much total RAM is on the host and how many agent processes or agent jobs typically run on it? • Which agent features/plugins are enabled (e.g., noise cancellation, VAD, turn detector, local ML models)? • Does memory usage climb gradually over time or spike during events (call start, high concurrency)? • Any recent changes deployed before this started (new plugin, config, model, or version upgrade)? Short-term steps you can take now • If the agent process is using excessive memory and you need immediate relief: restart the agent process during a low-traffic window or drain it gracefully if configured. • Temporarily disable heavy features (noise cancellation, model-based tools) on that host to see if memory usage drops. • If the process steadily grows over time, collect memory metrics over several hours to spot a leak pattern. If you share the command outputs above and answers to the questions, I’ll analyze them and recommend precise next steps: whether this is expected for your workload, immediate mitigations, and longer-term fixes (sizing, autoscaling, prewarming, or investigating leaks).
f
[no preview available]
t
Thanks — I appreciate the details you’ve already shared. Since there’s still ~9 GB free on the host, the next step is to focus on the agent process itself so we can tell if the warning is expected or indicates a problem. Please run the following (or paste the outputs if you already have them) so I can analyze the agent-level memory usage: 1. Top memory consumers: ◦ 
ps aux --sort=-%mem | head -n 10
1. Snapshot of system and processes: ◦ 
top -b -n 1 | head -n 20
1. Memory map for the agent process (replace <PID> with the PID from step 1): ◦ 
pmap <PID> | tail -n 20
1. Recent agent logs around the warning (replace 
<agent-service>
with your service name or use your agent log file): ◦ 
journalctl -u <agent-service> --since "1 hour ago" | grep -i memory
◦ or search the agent log file for 
memory
/ 
WARNING:livekit.agents
And please confirm these context items: • Total RAM on the host and instance type (e.g., 16 GB, AWS m5.large). • How many agent processes or agent jobs usually run on this host at peak. • Which heavy features/plugins are enabled (noise cancellation, VAD, LiveKit turn detector, local ML models). • Pattern: Does memory climb over time (possible leak) or spike during starts/concurrency peaks? • Any recent changes (new plugin, model, config, or version upgrade)? Short-term actions you can take right now • If the offending agent process is clearly the top memory consumer and you need immediate relief: restart or gracefully drain that agent during a low-traffic window. • Temporarily disable heavy plugins (noise cancellation, model-based tools) on this host to see if memory use drops. • If you’re on burstable instances, consider switching to dedicated instance types to avoid CPU throttling that can affect process behaviour. Once you paste the command outputs and answers to the context questions, I’ll: • Point out whether the memory use looks normal for your workload or suggests a leak. • Give a concrete, prioritized mitigation plan (restart/drain actions, config changes, sizing/autoscale recommendations). • Suggest longer-term monitoring and fixes (prewarming models, observability tweaks, or targeted investigation steps). You’re on the right track — share those outputs and I’ll dig in and give precise next steps.
8 Views
Open in Slack
PreviousNext
Powered by