We are experiencing connection timeouts during interactions with the avatar, occurring approximately 50% of the time. As a result, the agent often disconnects before the user finishes the session.

Thanks for reaching out, Ashish. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

{id: "AE_36b3H4dmBpTC" project_id: "p_1iotaaam2za" room_id: "RM_uwudRnx2hXFn" timestamp: "2026-03-26T201111Z" type: "Participant left (CONNECTION_TIMEOUT)" participant_id: "PA_fTSRDrbcVKZf" participant_identity: "agent-AJ_A2msC3k7UqXJ" node_id: "NM_OASHBURN1B_BP57mrMMMnJH" timestamp_us: 1774555871568164 disconnect_reason: "CONNECTION_TIMEOUT" }

Thanks — I received the event you pasted. That disconnect entry is helpful. What I’m responding to • The Participant left event for participant_id

PA_fTSRDrbcVKZf

(identity

agent-AJ_A2msC3k7UqXJ

) in room

RM_uwudRnx2hXFn

showing

disconnect_reason: "CONNECTION_TIMEOUT"

at 2026-03-26T201111Z on node

NM_OASHBURN1B_BP57mrMMMnJH

. What this event indicates • A

CONNECTION_TIMEOUT

means the participant (your avatar agent) failed to maintain the necessary keep-alives (ICE/STUN or transport heartbeats) and the server closed the session. This aligns with common causes such as network interruptions, TURN/ICE problems, or the avatar worker failing to send timely media or keep-alives. Targeted next checks — please run these and share results 1. Correlate server/client logs at the same timestamp ◦ Provide client SDK logs and avatar worker logs from ±30s around 2026-03-26T201111Z. ◦ Include any disconnect reason strings, ICE state transitions, or error stack traces. 1. ICE / connection-state events ◦ Capture

IceConnectionStateChanged

and

ConnectionStateChanged

events from the client and avatar worker. Do you see ICE going to

disconnected

or

failed

before the timeout? 1. WebRTC stats ◦ If possible, capture a short WebRTC getStats report from the client and from the avatar worker during a failing session. Look for packet loss, RTT spikes, or SRTP fallback. 1. Network checks ◦ Run continuous ping/traceroute between avatar worker ↔️ LiveKit server and client ↔️ LiveKit server for a few minutes during a repro. Note packet loss or large latency spikes. 1. Avatar worker health and deployment ◦ Share the instance type (e.g., t3/t4 vs m5/c5), CPU/GPU utilization at the time, and whether autoscaling or concurrency limits were in effect. ◦ Confirm whether the avatar server publishes directly into the room and whether it uses TURN. 1. Session configuration ◦ Share

user_away_timeout

value and any VAD/timeout settings applied to the AgentSession. ◦ Confirm whether you use any custom reconnection or ICE-restart logic in your client/worker. Quick mitigations to try immediately • Temporarily increase

user_away_timeout

to avoid premature closing while diagnosing. • If the avatar worker is on a burstable instance (t3/t4), switch to a non-burstable instance (m5/c5) for a test run to rule out CPU throttling. • Enable verbose ICE/logging on client and avatar worker and run a short repro test. What to send me now • Client SDK version and avatar worker version. • Avatar worker instance type and region, and whether TURN is configured. • The client and avatar worker logs for ±30s around 2026-03-26T201111Z (copy/paste or attach). • Any ICE/connection-state events or getStats output you captured. Once I have those logs and details I’ll: • Look for the exact ICE/transport state transitions leading to the CONNECTION_TIMEOUT. • Point out specific log lines that indicate root cause (network drop, ICE failure, CPU overload). • Recommend targeted fixes (config changes, deployment sizing, or reconnection handling) you can apply and test. I know frequent timeouts are disruptive — send the logs and environment details above and I’ll dig in with concrete next steps. , with interactive elements

{ 'attributes' : { 5 items 'code.filepath' : '/.uv/.venv/lib/python3.12/site-packages/livekit/agents/ipc/supervised_proc.py' , 'code.lineno' : 339 , 'code.function' : '_pong_timeout_co' , 'pid' : 209 , 'job_id' : 'AJ_A2msC3k7UqXJ' , }, }

Thanks — this new log snippet is useful. I’m responding to the supervised process ping timeout info you pasted (file

supervised_proc.py

, function

_pong_timeout_co

, job_id

AJ_A2msC3k7UqXJ

). That points to the avatar worker’s internal supervised process not responding to IPC pings, which can cause the worker to stop answering keep-alives and lead to CONNECTION_TIMEOUT disconnects. What this suggests • The avatar worker’s supervisor detected a pong timeout from a child process (the agent/job

AJ_A2msC3k7UqXJ

), meaning the worker’s internal IPC loop did not respond in time. • This is consistent with CPU/GPU throttling, long blocking operations, or the process crashing/hanging — all of which can prevent timely keep-alives and cause the LiveKit server to time out the participant. Targeted checks to run now 1. Collect the surrounding avatar worker logs ◦ Provide full log lines ±60s around the

_pong_timeout_co

event in

supervised_proc.py

. Include any stack traces, crash dumps, or repeated timeout lines. 1. Check process health and resource metrics ◦ CPU and GPU utilization at the time of the timeout. ◦ Memory usage and any OOM or throttling events. ◦ Whether the worker runs on a burstable instance type (t3/t4) and if CPU credits were exhausted. 1. Inspect IPC and worker thread activity ◦ Look for blocked threads, long-running synchronous tasks, or external calls (e.g., heavyweight model inference, network calls) that could block the IPC loop. ◦ Confirm whether the avatar generation path (audio → ByteStream → publish) is CPU/GPU-bound and whether batching or synchronous waits are used. 1. Check for process restarts or crashes ◦ Do you see the child process exiting, being respawned, or the supervisor repeatedly timing out and restarting the job? 1. Correlate with LiveKit connection events ◦ Match the timestamp of the

_pong_timeout_co

to the CONNECTION_TIMEOUT event you previously shared (2026-03-26T201111Z). Are they the same or within seconds? This will confirm causality. Immediate mitigations to reduce disconnect rate • Run the avatar worker on a non-burstable instance (m5/c5) for a test to rule out CPU throttling. • If heavy synchronous work is performed on the main IPC thread, move it to a worker thread or async queue so the supervisor can continue responding to pings. • Add temporary health checks and verbose logging around the IPC heartbeat to capture what the child process is doing when the pong is missed. • Increase heartbeat/pong timeout temporarily (only for testing) to see if transient stalls are the cause while you gather logs. What to send me next • Full avatar worker log excerpt ±60s around the

_pong_timeout_co

entry (include PID 209 context and any stack traces). • CPU/GPU and memory metrics for the worker at the time of the timeout. • Whether the worker instance is burstable (t3/t4) or non-burstable (m5/c5), and any autoscaling/concurrency configuration. • The avatar worker version and any recent code changes to the agent pipeline. • Confirmation of whether the

_pong_timeout_co

timestamp aligns with the CONNECTION_TIMEOUT event (2026-03-26T201111Z). What I’ll do once you send those • Identify whether the pong timeout was due to blocking work, resource exhaustion, or a crash. • Point to exact log lines that indicate the root cause. • Recommend specific code/architecture changes (e.g., offloading blocking work, increasing worker size, IPC tuning) and configuration tweaks to stabilize sessions. You’re on the right track — the ping timeout is a strong lead. Send the logs and metrics above and I’ll help you pinpoint the fix and test mitigations. , with interactive elements