Thanks — I can see from the log you shared that a worker job for room_id=RM_WAFqZ5NbMDBB became unresponsive, the agent sent SIGUSR1, then killed the process, and the process exited with code -10. I’ll help you narrow down the root cause and next steps to investigate. First, a few clarifying questions so I can give targeted guidance: • Is this a self-hosted deployment or LiveKit Cloud? • What instance type are you running the agent on (for example AWS t3/t4g, c5, m5, etc.)? • Was this a one-off incident or are you seeing this repeatedly? If repeated, how often and under what load (approx concurrent voice-agent sessions)? • Do you have resource metrics available (CPU, memory, disk, and for AWS: CPUCreditBalance) around the timestamps in the log (2026-03-18 1356–1357)? • Do you have Agent Observability enabled for these sessions (so we can review traces, audio playback, transcripts)? While you gather that info, here are immediate things to check and actions you can take to diagnose and reduce recurrence: 1) Check instance CPU/memory and burst credits • If you’re on burstable instances (t3/t4g), check AWS CloudWatch for CPUCreditBalance around the incident time. Exhausted credits lead to heavy CPU throttling which can make processes unresponsive. • Even on non-burstable instances, look at CPU and memory spikes that coincide with the timestamp. 2) Review worker capacity and concurrent jobs • The log shows “worker is at full capacity, marking as unavailable” shortly before the unresponsive message. Confirm concurrent job count at that time — high concurrency or resource-heavy sessions (STT + LLM + TTS + WebRTC) can exhaust resources. • If you’re hitting capacity regularly, consider increasing instance size or service replicas. 3) Gather detailed logs for the killed process • Collect the job’s stdout/stderr and agent logs immediately before the SIGUSR1 and kill. Look for LLM or STT timeouts, long blocking calls, or repeated errors. • Search for long GC pauses, blocking I/O, or stack traces that might indicate deadlocks or infinite loops. 4) Correlate session-level behavior • If Agent Observability is available, inspect the affected room/session for the exact turn or model call that preceded the hang (e.g., a long-running LLM request or TTS generation). • Look for increased TTFT/TTFB or tool/LLM latency spikes that match the timeframe. 5) Reproduce under controlled load • If possible, run a single failing session in isolation to see if the job still becomes unresponsive. Then increase concurrency to reproduce load-related failures. • This helps separate a deterministic bug (infinite loop, deadlock) from resource exhaustion. 6) Short-term mitigation • If you confirm CPU throttling or resource exhaustion, move agents to non-burstable instances (c5/m5 family) or scale horizontally to reduce concurrency per-agent. • Tune agent

load_threshold

and

load_fnc

(server options) to prevent new jobs from being scheduled when the worker is nearing resource limits. Notes about the exit code and signals • The log shows exit code -10 and that the worker sent SIGUSR1 then killed the process for being unresponsive. Exit code -10 isn’t documented here specifically, but non-zero exit codes generally indicate abnormal termination (crash, fatal error, or signal). The agent’s kill sequence means the job stopped responding to signals — often symptomatic of CPU starvation, deadlocks, or blocking native calls. If you can share the answers to the clarifying questions above and any additional logs (process stdout/stderr surrounding 1356–1357), I’ll help you interpret them and provide the next concrete steps — for example, which metrics to pull, what log lines to look for, or what configuration changes to make. Sources: Diagnosing Connection Errors with Connection Test Utility | Job lifecycle | LiveKit Documentation | Server options | LiveKit Documentation | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Self-hosted deployments | LiveKit Documentation, with interactive elements

Thanks — this is very helpful context. You’re on LiveKit Cloud, running agents on c6g.large, it’s happened more than once, you saw CPU spikes, Agent Observability shows no audio (likely because the process died before it could record) — I’ll summarize what’s most likely happening, and give a focused, actionable investigation plan and mitigations you can run right now. Short summary of what the logs show • A per-room job for room_id RM_WAFQZ5NMD8B became unresponsive; the agent sent SIGUSR1 and then killed the process, which exited with code -10. • The worker hit full capacity just before the hang and was later marked available again after the kill. • STT and TTS calls were active immediately before the hang, but Agent Observability has no audio for that session — consistent with the process dying before audio could be flushed. Most likely causes • CPU saturation or temporary CPU starvation during inference/processing causing the job to stop responding (you reported CPU spikes). Even on non-burstable instances, sustained 100% CPU on a process can make it miss health checks and signal handlers. • Heavy per-job work (STT/LLM/TTS + WebRTC) combined with worker concurrency exhausting the agent’s capacity. • A blocking/native call, deadlock, or long GC pause inside the job process that prevented it from responding to signals or flushing audio before termination. What I recommend you do next (ordered: quick checks first, then deeper diagnostics) 1) Share two small items so I can point to exact logs to inspect • Exact timestamps of the incident(s) you want investigated (UTC or your local timezone). • Any other room_id(s) for recent occurrences (you’ve provided RM_WAFQZ5NMD8B already). 2) Quick checks you can run now • Pull instance CPU utilization and per-process CPU% around the timestamps (one-minute resolution or better). Confirm whether CPU was at or near 100% when the kill occurred. • Check system load and memory use at the same times. Look for sustained high load or OOM events. • If you have CloudWatch (or equivalent), check for any system-level throttling/interrupts during the window. 3) Correlate Observability and agent logs • With the timestamps + room_id I can tell you the exact agent log time ranges to export. Look in agent logs for lines immediately before the “process is unresponsive” message for: ◦ Repeated timeouts/long latencies from STT/LLM/TTS calls ◦ Any stack traces, uncaught exceptions, or native library errors ◦ Long GC pauses or “blocking” messages if your runtime logs them • In Agent Observability, check STT/LLM/TTS latencies and TTFT/TTFB on the last successful turn. A very long model or TTS call can indicate the operation that blocked the process. 4) Capture richer diagnostics for one reproduction (short window) • Enable more verbose agent/job logging for a short time and capture stdout/stderr for the child job processes. • If possible during a test window, enable core dumps or collect a process stack trace when the process becomes unresponsive (this will reveal deadlocks or native blocking). I can tell you the exact commands/patterns to capture these based on the runtime. 5) Short-term mitigations to reduce failures now • Reduce concurrency per agent: lower the worker

load_threshold

so the worker rejects new jobs earlier and avoids hitting full capacity. • Scale horizontally: add agent replicas so fewer jobs are scheduled per process. • If you see consistent CPU saturation, move to larger instances (more vCPUs) to give headroom for spikes. 6) Longer-term changes to consider • If frequent heavy inference is expected, prefer instance families with predictable CPU performance and more cores (e.g., m5/c5/c6i families) rather than constrained machines. • Add autoscaling for agent replicas driven by CPU or custom load metrics so the system reacts automatically to spikes. • Keep Agent Observability enabled and use it to correlate failures with model or tool calls. What I can do for you next • If you share the exact timestamps (and any additional room_ids) for the recent incidents, I will: ◦ Tell you the exact agent-log ranges to export and which log lines to look for. ◦ Point to the Observability trace fields and latency metrics to inspect for that session (STT/LLM/TTS latencies, TTFT/TTFB). ◦ Provide the exact diagnostic commands or config changes to enable stack traces or core dumps for a controlled test. • If you prefer, I can first walk you through exporting CPU and instance metrics so we confirm whether CPU saturation is the immediate trigger. Which would you like to do next — share the timestamps/room_ids for me to analyze logs, or walk through exporting CPU metrics and enabling more verbose logs first? Sources: Terms of Service | LiveKit | Streamline troubleshooting with Agent Observability | LiveKit | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Terms of Service | LiveKit | Collect recording consent with tasks | LiveKit Documentation, with interactive elements