https://livekit.io logo
Join Slack
Powered by
our self hosted livekit server and agents are on d...
# ask-ai
a
our self hosted livekit server and agents are on different servers, sometimes the agent's connection timesout, it happens like once a day, is there any reason why this could happen randomly? we get this error: 
connectionError: engine: connection error: could not establish publisher connection: timeout"
It connects again when we restart the agents
t
Thanks for reaching out, Taha. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m sorry you’re seeing those intermittent timeouts. I’ll help troubleshoot why agents sometimes hit 
connectionError: engine: connection error: could not establish publisher connection: timeout
and recover only after a restart. Quick acknowledgement: you said your LiveKit server and agents run on different servers and the agent’s connection times out roughly once a day; restarting the agent fixes it temporarily. Below are the most likely causes and targeted steps to diagnose and resolve them. Possible causes and what to try next 1) Network / firewall / NAT issues (most common) • Why: WebRTC requires steady ICE/STUN/TURN traffic and keep‑alives. Firewalls, NAT timeouts, or proxies can drop or block UDP or later packets, causing negotiation to time out. • What to do: ◦ From the agent host, run the LiveKit Connection Test to verify WebSocket, WebRTC, and TURN connectivity: https://livekit.io/connection-test ◦ If UDP is unreliable or blocked, force TURN-only connections by setting 
iceTransportPolicy: 'relay'
in your agent/client RTC config so traffic stays on relays over TCP/443. ◦ Check network devices between agent and server for connection tracking / NAT timeouts and increase keepalive intervals if possible. 2) Proxy / deep packet inspection (ZScaler or similar) • Why: Proxies doing deep packet inspection or protocol detection can delay or modify WebRTC/TURN traffic and cause timeouts. • What to do: ◦ If you use ZScaler (or similar), add bypass rules for your LiveKit domains and TURN hosts so the proxy doesn’t inspect those connections. Example rule pattern: ▪︎ url.domain=turn.livekit.cloud detect_protocol(no) ▪︎ url.domain=<your-project>.livekit.cloud detect_protocol(no) ◦ Re-run the Connection Test after applying bypasses. 3) Agent or server resource exhaustion / burstable instances • Why: If agents or the LiveKit host run on burstable instances, CPU throttling or other resource spikes can slow negotiation or processing and cause timeouts even when average CPU looks low. • What to do: ◦ Avoid burstable instance types (t3/t4g) for production agents or LiveKit servers; use non‑burstable types (for example, m5/c5 families). ◦ Monitor CPU, memory, and i/o around the timeouts. If on AWS, check 
CPUCreditBalance
for burstable instances. 4) Agent process state / teardown problems • Why: If the agent becomes unresponsive or crashes without cleanly leaving a room, subsequent connections can fail until the agent is restarted. • What to do: ◦ Add robust health checks and auto‑restart (systemd, k8s liveness probes) so the agent automatically restarts if it becomes unhealthy. ◦ Implement retry logic with exponential backoff when establishing publisher connections to handle transient failures. ◦ Log and capture the agent’s stack traces / logs around the timeout event to spot hangs or deadlocks. 5) Deployment / redundancy considerations • Why: Single agent workers or colocated workloads increase the blast radius of transient issues. • What to do: ◦ Run multiple agent workers per logical agent_name for failover. ◦ Place agents on dedicated hosts (or isolated pods) to avoid noisy-neighbor interference. Short checklist to run now 1. Run the LiveKit Connection Test from the agent machine: https://livekit.io/connection-test 2. If UDP/blocking is detected, configure 
iceTransportPolicy: 'relay'
on the agent. 3. Verify no proxy (ZScaler) is inspecting LiveKit/TURN traffic; add bypass rules if needed. 4. Check agent/server instance types and CPU credit usage; move off burstable instances if used. 5. Enable health checks and automatic restarts for agents; add retries with exponential backoff. If you’d like, tell me: • Are agents behind a corporate proxy or ZScaler? • Are you using TURN (and which TURN host)? • What instance types your agent and LiveKit servers run on? • Any agent logs (timestamps around the timeout) you can paste here? With those details I can suggest the most targeted next steps and help interpret connection test results. Sources: Understanding the STATE_MISMATCH Disconnect Reason | Firewall Tips for LiveKit Connectivity | LiveKit | Configure ZScaler proxy for LiveKit connections | LiveKit | Diagnosing Connection Errors with the Connection Test Utility | LiveKit | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS
a
how can I run LiveKit Connection Test on my server? I can only ssh into it, there is no UI
t
Thanks — good question. You can’t run the full LiveKit Connection Test from a headless server over SSH because the test requires a browser WebRTC stack (and optional mic/camera) to fully exercise signaling, ICE, STUN/TURN and publish flows. However, you can do useful command-line checks from your SSH session to diagnose the network and service-level issues, and you can run the complete Connection Test from any client browser. What you can’t run from SSH • Browser WebRTC checks (ICE negotiation, gather candidates) • Publish audio/video checks (need microphone/camera) • The interactive TURN/UDP vs TCP measurements What you can do from the server (use these to narrow problems) 1) Verify TLS / WebSocket reachability to your LiveKit host • Run a TLS handshake test to confirm TCP/443 and TLS work:
Copy code
echo -e "GET /rtc/connect HTTP/1.1\r\nHost: <http://YOUR-LIVEKIT-SERVER.com|YOUR-LIVEKIT-SERVER.com>\r\n\r\n" | timeout 5 openssl s_client -connect <http://YOUR-LIVEKIT-SERVER.com:443|YOUR-LIVEKIT-SERVER.com:443> -servername <http://YOUR-LIVEKIT-SERVER.com|YOUR-LIVEKIT-SERVER.com>
• A successful TLS handshake shows WSS/HTTPS reachability. 2) Test TURN endpoint reachability • UDP test (may show blocked UDP):
Copy code
nc -uz your-turn-domain.turn.livekit.cloud 3478
• TLS/TCP test:
Copy code
timeout 5 openssl s_client -connect your-turn-domain.turn.livekit.cloud:443 -servername your-turn-domain.turn.livekit.cloud
• If UDP is blocked but TCP/443 works, forcing TURN-over-TCP (relay) will be required for clients. 3) Check service logs and process health • LiveKit, proxy, or load-balancer status:
Copy code
systemctl status livekit
  journalctl -u livekit --since "1 hour ago" | tail -n 200
  tail -n 200 /var/log/nginx/error.log
• Look for errors around the time the agent timed out. 4) Capture traffic to observe negotiation attempts • Use tcpdump to see whether clients or agents are reaching the server and whether packets stop:
Copy code
sudo tcpdump -i any host AGENT_IP or host <http://YOUR-LIVEKIT-SERVER.com|YOUR-LIVEKIT-SERVER.com> -w /tmp/conn.pcap
• Inspect the capture with Wireshark on your workstation (transfer the pcap). 5) Basic connectivity checks • Ping and traceroute to see routing/NAT issues:
Copy code
ping -c 5 CLIENT_IP
  traceroute CLIENT_IP
• Check for firewall rules:
Copy code
sudo iptables -L -n -v
  sudo ufw status verbose
When to run the full Connection Test • Open a browser on a client machine (not the headless server) and go to: https://livekit.io/connection-test • Run the full suite (WebRTC, TURN, publish tests). This will show whether UDP, TURN, or WebSocket signaling problems exist from the client network. Short fixes to try quickly • If UDP is blocked on the client network, configure clients to use TURN only: ◦ Set 
iceTransportPolicy: 'relay'
in the client/agent RTC config. • If a corporate proxy (e.g., ZScaler) is present, add bypasses for your LiveKit and TURN domains so the proxy doesn’t inspect or break WebRTC/TURN traffic. • Add health checks and auto-restart for agents so a hung agent is automatically recycled. If you want, share: • The LiveKit/TURN hostnames you want to test (I’ll tell you exact openssl / nc commands to use), and • One timestamp when the agent timed out and the agent’s IP — I can point you to exactly which logs and pcap filters to capture. Sources: Running LiveKit on AWS | Diagnosing Connection Errors with Connection Test Utility | Firewall Tips for LiveKit Connectivity | LiveKit | Diagnosing Connection Errors with the Connection Test Utility | LiveKit
2 Views
Open in Slack
PreviousNext
Powered by