getting this error suddenly on an agent deployed o...
# ask-ai
s
getting this error suddenly on an agent deployed on livekit cloud. It was working till a few days back "level":40,"time":1766758050607,"pid":273,"hostname":"deployment-p-2pd40f9yqi0-ca-3w3cp2bue6d7-5f8f4bbd6c-nws9k","tts":"inference.TTS","attempt":3,"error":{"body":null,"retryable":true,"name":"APIConnectionError"},"msg":"failed to synthesize speech, retrying in 2000s"} {"level":40,"time":1766758052183,"pid":46,"hostname":"deployment-p-2pd40f9yqi0-ca-3w3cp2bue6d7-5f8f4bbd6c-nws9k","memoryUsageMB":564.91015625,"memoryWarnMB":500,"memoryLimitMB":0,"msg":"process memory usage is high"} [SESSION ERROR] APIConnectionError: Error connecting to LiveKit WebSocket at WebSocket.onError (file:///app/node_modules/.pnpm/@livekit+agents@1.0.27_@livekit+rtc-node@0.13.20_zod@3.25.76/node_modules/@livekit/agents/dist/inference/utils.js2916) at Object.onceWrapper (nodeevents634:26) at WebSocket.emit (nodeevents519:28) at emitErrorAndClose (/app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/websocket.js104113) at process.processTicksAndRejections (nodeinternal/process/task queues90:21) { body: null, retryable: true }
t
Thanks for reaching out, Shubh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I see two things in the logs you shared: repeated
APIConnectionError
when the agent tries to reach the LiveKit WebSocket, and a high memory usage warning (≈565 MB, above the 500 MB warning). I’ll help you troubleshoot both since they can be related. Quick checklist and steps to try 1. Verify basic network/connectivity • Can the host/container reach LiveKit Cloud endpoints? Run a simple connectivity test from the same environment (DNS resolution + TCP to the LiveKit signaling endpoint). If you have the Connection Test utility, run it from the agent host. • Check for recent network changes (VPC, firewall, proxy, egress rules, or NAT gateway changes) or cloud infra updates that could block outbound WebSocket/TCP. 1. Inspect WebSocket stability • Confirm the agent is pointed to the correct LiveKit Cloud URL and using the expected API key/credentials. • Check for intermittent packet loss or long latency from the host to LiveKit (mtr/traceroute, ping). WebSocket errors often follow degraded network quality. • Look at server-side LiveKit logs (if available in your dashboard) for any rejected connections or auth/limit errors. 1. Address high memory usage • Review what the agent is doing when memory climbs (TTS synthesize attempts, audio buffers, concurrent jobs). High memory can cause event loop stalls or process restarts which break connections. • If you’re running in a container/orchestrator, confirm memory limits and whether the process is being OOM-killed or throttled. • Reduce memory pressure by lowering concurrency, clearing/streaming buffers sooner, or using smaller TTS models if configurable. 1. Skim SDK/runtime issues • Ensure the deployed agent version (you’re using @livekit/agents v1.0.27 per the stack trace) is compatible with your environment and there were no recent forced upgrades/changes. If you recently updated the runtime or SDK, roll back to the previously-working version to test. Questions to narrow this down • Did anything change in your deployment around when this started (network rules, cloud region, image updates, autoscaling config)? • Are all agents in the same deployment affected, or only this host/pod instance? • Do you see any OOM/kube eviction events, container restarts, or platform-level network issues at the same timestamps as the errors? • Can you run a short connectivity test (ping/traceroute/mtr) from the failing host and paste results or key findings? Next steps I can take for you • Walk through a connection test command to run in your environment and interpret results. • Suggest memory-tuning changes or a quick diagnostic to capture a heap snapshot or process metrics during TTS attempts. • If you share whether this is a single node or multiple, I can propose targeted mitigations (scale down concurrency, increase memory, or pin to a stable agent image). Tell me which of the above you want to start with and provide the answers to the questions you can — I’ll guide you step-by-step. Sources: How to Enable Agent-to-Agent Speech Detection in LiveKit | Diagnosing Connection Errors with Connection Test Utility | Agent server options | LiveKit docs