I am see high CPU usage on my Livekit agent. I am running on t3. xlarge (4 cpu cores & 16MB memory)

Thanks for reaching out, Deepak. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Getting this error: Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/asyncio/tasks.py\", line 520, in wait_for\n return await fut\n ^^^^^^^^^\n File \"/app/.venv/lib/python3.12/site-packages/livekit/agents/ipc/channel.py\", line 47, in arecv_message\n return _read_message(await dplx.recv_bytes(), messages)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/app/.venv/lib/python3.12/site-packages/livekit/agents/utils/aio/duplex_unix.py\", line 35, in recv_bytes\n len_bytes = await self._reader.readexactly(4)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/asyncio/streams.py\", line 752, in readexactly\n await self._wait_for_data('readexactly')\n File \"/usr/local/lib/python3.12/asyncio/streams.py\", line 545, in _wait_for_data\n await self._waiter\nasyncio.exceptions.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/app/.venv/lib/python3.12/site-packages/livekit/agents/ipc/proc_pool.py\", line 167, in _proc_spawn_task\n await proc.initialize()\n File \"/app/.venv/lib/python3.12/site-packages/livekit/agents/ipc/supervised_proc.py\", line 169, in initialize\n init_res = await asyncio.wait_for(\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/asyncio/tasks.py\", line 519, in wait_for\n async with timeouts.timeout(timeout):\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/asyncio/timeouts.py\", line 115, in __aexit__\n raise TimeoutError from exc_val\nTimeoutError

Thanks — I’m responding to the traceback you pasted (asyncio CancelledError → TimeoutError during agent IPC/init). That traceback most often appears when the agent’s subprocess initialization or IPC read is being delayed long enough to hit the initialization timeout. Likely cause • Your t3.xlarge is burstable. If CPU credits are exhausted the instance is heavily throttled, causing normally-fast operations (IPC reads, model inferences) to exceed the init timeout and raise the shown TimeoutError. Immediate checks 1. In CloudWatch check

CPUCreditBalance

for the instance during the error window. If it’s near zero, throttling is happening. 2. Check agent logs around the same timestamps for inference latency spikes, queuing, or repeated initialization attempts. 3. Confirm how many agents/processes you run on that instance and current concurrency—oversubscription can cause similar timeouts. Short-term mitigations • Enable

t3.unlimited

(if acceptable for cost) to avoid immediate credit exhaustion. • Reduce agent concurrency on that host or move some agents off the instance. • Increase the IPC/init timeout temporarily if you need more time while fixing capacity (only as a stopgap). Recommended long-term fixes • Move agents to non-burstable compute-optimized instances (for example

c5.large

,

c6i.large

, or

m5.xlarge

) to avoid CPU credit throttling and get stable latency. • Scale horizontally (more agents on separate instances) so each process has predictable CPU headroom. To give a precise remediation plan, please share: • Agent type (voice or video) • Models in use (STT/TTS/VAD/LLM/ONNX inference) • Number of agents/processes on the t3.xlarge and typical concurrent sessions • CloudWatch

CPUCreditBalance

around the error time (or tell me if it was near zero) • Any recent config/traffic changes and a short snippet of the agent log around the timeout Send those details and I’ll recommend exact instance sizing, timeout config changes, and next steps. Sources: Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Deploy and scale agents on LiveKit Cloud

CPUCreditBalance was enough

Thanks — since CPU credits look sufficient, let’s narrow this down. Could you provide the following so I can pinpoint the cause of the TimeoutError? • Agent type: voice or video? • Models or features in use: STT, TTS, VAD, ONNX, LLMs, etc. • How many agent processes are running on the t3.xlarge and how many concurrent sessions per agent? • Recent traffic/config changes around when the error started. • A short log snippet (10–20 lines) from the agent immediately before and after the timeout. • CloudWatch CPU% and system load (1m/5m) around the error timestamps. • Whether you’ve increased IPC/init timeouts or are using defaults. With those details I’ll propose targeted next steps (process-level debugging, timeout tuning, instance sizing, or workload changes).

Agent type: voice STT, TTS, VAD,BVC 2 boxes of t3.xlarge. 20 concurrent conversations overall. CloudWatch CPU% = 65%

Thanks — I’m responding to your details: voice agent using STT/TTS/VAD/BVC, 2 × t3.xlarge, ~20 concurrent (≈10 per box), CloudWatch CPU ~65%. Given that, next steps to pinpoint and resolve the TimeoutError: 1. Confirm per-instance process/concurrency • How many agent processes run on each t3.xlarge? • Do conversations spawn separate processes or share a single agent process? 1. Measure real-time latency and backpressure • Capture inference latencies for STT/TTS/VAD/BVC (50–160ms expected; >500ms indicates resource problems). • Provide a short agent log snippet (10–20 lines) around a TimeoutError occurrence. 1. Inspect system metrics around the error • Share the 1m/5m system load averages and CPU% for the instance at the error time (CloudWatch). 1. Short-term mitigations you can apply immediately • Temporarily reduce concurrent sessions per instance (move some conversations to the other box or pause connections) to see if timeouts stop. • If you must keep current capacity, increase the IPC/init timeout as a stopgap (only temporary). • Consider enabling t3.unlimited to avoid sudden throttling (monitor cost). 1. Strong recommendation (long-term) • Move agents to non-burstable compute‑optimized or general‑purpose instances (for example, c5.large, c6i.large, or m5.xlarge) so model inference and IPC aren’t affected by credit throttling. If you can provide the answers/logs from items 1–3, I’ll: • Determine if this is capacity vs. software/configuration, • Recommend exact instance sizing or concurrency limits, • Give precise timeout or config changes to apply. Sources: Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | examples/drive-thru | livekit/livekit_composite | DeepWiki

what does Livekit do when agents are running at full capacity, and hence unavailable and a new call request is submitted

Thanks — I’m responding to your question about what happens when all agents are at full capacity and a new call arrives. Short answer • LiveKit will not queue the call. If no agent is available, the dispatch fails and the call request is rejected (fail‑fast). What that means in practice • Agents report capacity and stop accepting jobs when their load exceeds configured thresholds. New calls are not assigned and the caller receives a failure rather than being held in a LiveKit queue. • During deployments or shutdowns agents can drain (finish existing sessions) but still will not queue new calls. How to handle this • Run multiple agent workers per service for redundancy and load distribution. • Implement external queuing or fallback logic in your application (e.g., voicemail, callback scheduling, retry with backoff). • Use LiveKit Cloud if you want automatic scaling to add capacity during spikes. • Tune agent capacity settings (

load_fnc

/

load_threshold

) to better reflect your environment. If you want, tell me which fallback behavior you prefer (queue, voicemail, retry) and I’ll outline a simple architecture and implementation options. Sources: Agents Framework | livekit/livekit_composite | DeepWiki | Server lifecycle | LiveKit docs | Agent server options | LiveKit docs | Deploy and scale agents on LiveKit Cloud | Deploy and scale agents on LiveKit Cloud

how many concurrent calls can a 4cpu core machine handle

Thanks — I’m responding to your question about how many concurrent calls a 4‑CPU machine can handle. Short answer • For voice agent workloads with STT/TTS/VAD/BVC, expect roughly 5–10 concurrent calls per 4‑vCPU machine under typical conditions. Heavy models or extra features push you toward the lower end. Key caveats • Burstable instances (t3) are unreliable for sustained load even if CPU% looks moderate; use non‑burstable compute‑optimized instances for predictable capacity. • Average CPU% (e.g., 65%) can hide short spikes that cause timeouts; monitor per-request inference latency (target 50–160ms; >500ms is problematic). If you want a precise recommendation, tell me: • Exact models in use and whether inference runs locally (ONNX/LLM) • How many agent processes per host and whether sessions are pinned to processes • Target latency/SLA and acceptable failure/retry behavior With those I’ll give a concrete concurrency target and instance-sizing recommendation. Sources: examples/drive-thru | livekit/livekit_composite | DeepWiki | Benchmarking | LiveKit docs | Self-hosting the Ingress Service | LiveKit docs | Self-hosting the Egress Service | LiveKit docs

how to find this: How many agent processes per host

Good question — here are simple ways to find how many LiveKit agent processes are running on a host. 1) On a Linux host (processes) • Replace

agent.py

with your agent binary/script name: ◦ Count:

pgrep -f agent.py | wc -l

• Or list:

ps aux | grep agent.py | grep -v grep

2) If using containers • Docker:

docker ps --filter "name=agent" --format "{{.Names}}" | wc -l

• Kubernetes:

kubectl get pods -l app=livekit-agent --no-headers | wc -l

3) Use system monitors • Run

htop

or

top

and filter/search for the agent process name to see each process and its CPU/memory. 4) From agent logs • Start the agent with debug logging (

--log-level=DEBUG

) and inspect lifecycle logs — each job run creates a process and debug logs will show process spawn/exit events. Notes • Each agent job runs in its own process, so counting those processes gives the number of active agent jobs on the host. • For LiveKit Cloud, process-level counts aren’t exposed; use the Cloud dashboard metrics for session/worker counts. If you tell me your agent process name or whether you run in Docker/Kubernetes, I’ll give the exact command tuned to your setup. Sources: Agent server options | LiveKit docs | Server lifecycle | LiveKit docs | Deploy and scale agents on LiveKit Cloud

How many agent processes per host => 1 Sessions are pinned to host

Thanks — I’m responding to your confirmation that you run 1 agent process per host and sessions are pinned. Key implications • Because sessions are pinned to a single agent process on each host, that host is the only place those sessions can run. If the process is saturated, throttled, or crashes, those pinned sessions will fail and will not automatically fail over to another host. • Adding more hosts or agent processes will not provide failover for already-pinned sessions unless you remove pinning or implement session handoff logic. Immediate recommendations • Move off burstable t3 instances to non‑burstable compute instances (for example

c5.large

,

c6i.large

, or

m5.xlarge

) to avoid CPU-credit throttling and reduce timeouts. • If you must stay on t3, enable

t3.unlimited

and monitor

CPUCreditBalance

and per-request inference latency closely. • Remove strict pinning or implement session handoff if you want automatic failover. Options: ◦ Unpin sessions so dispatch can place jobs on healthy workers. ◦ Implement application-level handoff: detect host overload and transfer sessions (requires application logic and careful state sync). • Implement external queuing or retry logic for new calls — LiveKit does not provide built-in call queuing. Use a queue/dispatcher or backoff/retry in your caller flow to avoid immediate rejection when the pinned host is saturated. Monitoring to collect now • Per-request inference latencies (p50/p95/p99) for STT/TTS/VAD/BVC. • CPU% and 1m/5m load during TimeoutErrors. • Agent process logs around IPC/init timeouts. •

CPUCreditBalance

if still on t3. If you’d like, I can: • Recommend exact instance sizing for your 10 sessions/host and model mix, • Outline a simple architecture to remove pinning while preserving session affinity, • Provide commands and metrics to collect the latency and load traces. Which of those would you like next?