routing/localrouter.go:113 could not handle new participant "error": "request timed out"

Thanks for reaching out, Eray. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

We are running a self-hosted LiveKit server (v1.9.1) with a SIP service and a Python agent worker (livekit-agents 1.4.3) on a single VM with 4 CPU cores and 7.5GB RAM. Setup: • LiveKit server, SIP service, and agent worker all running on the same VM • Redis Sentinel on separate servers •

NUM_IDLE_PROCESSES=10

,

LOAD_THRESHOLD=1.0

•

dispatchRuleIndividual

with

roomPrefix: "call-"

Problem: When 10 concurrent SIP calls arrive simultaneously, only 1-2 agents connect successfully. The rest crash with: s

livekit.rtc.room.ConnectError: engine: signal failure: client error: 401 Unauthorized - no permissions to access the room

The pattern in worker logs is always:

signal connection failed on v0 path: Timeout("signal connection timed out")

1. Then

failed to connect: Signal(Client(401, "no permissions to access the room"))

, retrying 1/3, 2/3, 3/3 2. Final crash with 401 Server logs show: e

routing/localrouter.go:113  could not handle new participant  "error": "request timed out"
routing/signal.go:323  could not send signal message  "error": "request timed out"
service/utils.go:53  error handling request  {"status": 401, "path": "/rtc/validate", "error": "no permissions to access the room"}

Important observations: Single call works perfectly every time Problem only occurs under concurrent load (10 simultaneous calls) • CPU was at ~99.6% during the load test (

0.4% idle

) Questions: Can CPU saturation cause token validation to fail with 401? 1. Is there any configuration in

livekit.yaml

to increase signal processing timeout or queue size? 2. What is the recommended minimum CPU count for 10 concurrent SIP calls with agent workers on the same machine? 3. Should the agent worker always run on a separate machine from the LiveKit server?

Thanks — this is helpful detail. I’m responding to the timeout + 401 errors you’re seeing when 10 concurrent SIP calls arrive (routing/localrouter.go:113 timeouts, signal timeouts in worker logs, then “no permissions to access the room”). Short answers up front • Yes — CPU saturation at ~99% can cause the signal/authentication path to time out and lead to 401 Unauthorized errors. • No — there is no exposed setting in

livekit.yaml

to increase internal signal-processing timeouts or queue sizes. • For your workload, a single VM with 4 CPUs is undersized; the recommended minimum when co-locating server + agents for ~10 concurrent SIP calls is around 8 CPU cores. • It is strongly recommended to run agent workers on a separate machine from the LiveKit server to avoid resource contention. Why this is happening • The routing layer must process participant initialization and token validation during join. If the server is CPU bound, those request handlers can time out (you see “signal connection timed out” and “could not handle new participant … request timed out”). • When the initial join attempt times out, retries may occur after token validity windows or internal state changes, which results in the later 401 “no permissions to access the room” responses you see in the logs. • The single-call success and concurrent-call failures, combined with CPU ~99.6% during the test, strongly point to resource contention rather than a token generation bug. Actionable next steps 1. Short-term test to confirm resource contention ◦ Run a load test with the same 10 concurrent calls but throttle agent workers so joins are staggered (not all simultaneous). If failures disappear, it confirms concurrency + CPU saturation is the root cause. 1. Immediate mitigation options ◦ Move agent workers off the LiveKit server to a separate VM (even a small separate instance). This will free CPU for signal/auth and reduce timeouts. ◦ If you must keep them colocated temporarily, increase the VM to at least 8 CPU cores and monitor again. 1. Recommended deployment changes ◦ Dedicated host for LiveKit server(s) and a separate host (or autoscaled pool) for agent workers. ◦ Ensure Redis/network latency is low and the LiveKit server has headroom for spikes in join traffic. 1. Monitoring and validation ◦ During another load test, watch: ▪︎ CPU and load averages ▪︎ Request latencies for /rtc/validate and signaling endpoints ▪︎ Worker logs for signal timeouts ◦ Collect a small log snippet (10–20 lines) from the LiveKit server around the time of a failed join — earlier lines before the timeout often indicate which step timed out. Details I can help with next • If you want, I can help you: ◦ Choose a target VM size for your expected concurrency (I can recommend memory and CPU targets if you share expected simultaneous media tracks per call). ◦ Suggest a simple staggered join test script to confirm the hypothesis. ◦ Review a short server log excerpt (around the timeout) to point to the exact code path that timed out. If you’d like, send one representative server log block (10–20 lines) around a failed join and confirm whether you want a recommendation for an 8‑core VM size or a different target based on expected track counts; I’ll provide concrete sizing and test steps. Sources: Going beyond a single-core | Troubleshooting 401 Unauthorized and Token Errors | LiveKit | Self-hosted deployments | LiveKit Documentation

We currently run LiveKit server, SIP service, and agent worker all on the same VM (4 CPU, 7.5GB RAM). We are experiencing CPU saturation under 10 concurrent SIP calls. We want to separate these services across multiple VMs. Our questions: 1. Can LiveKit server and SIP service stay on the same VM, or should they also be separated? 2. Are there any network requirements between the agent worker VM and the L,iveKit server VM (open ports, WebSocket, firewall rules, etc.)? 3. For 10 concurrent SIP calls with a Python voice agent (OpenAI + Silero VAD), what is the recommended CPU/RAM for each VM separately?

Thanks — I’m responding to your three questions about separating services, network requirements between agent workers and the LiveKit server, and sizing for 10 concurrent SIP calls with Python voice agents (OpenAI + Silero VAD). 1) Should LiveKit server and SIP service stay on the same VM? • They can co‑exist on the same VM, but it’s not recommended at scale. The SIP service (media handling, possible transcoding) and the LiveKit SFU (signaling, participant init, routing) have different, often bursty CPU and network demands and compete for the same resources. • For your current symptom (CPU ~99% and timeouts/401s under 10 concurrent calls), separating them is the safer choice. If you must keep them together temporarily, the VM must be significantly larger than your current 4‑core instance (see sizing below). 2) Network requirements between agent worker VM(s) and LiveKit server VM • Agent workers establish outbound WebSocket (WSS) signaling connections to the LiveKit server (commonly port 443). Ensure the workers can open outbound TCP connections to the LiveKit host on the signaling port you expose (

<wss://your-livekit-host>

). No inbound ports are required on the worker VMs for signaling. • Media uses WebRTC (UDP) — ensure the network path between worker and LiveKit server allows UDP for media. By default LiveKit uses a UDP range (configurable via

rtc.udp_port_range

); make sure firewalls/NACLs allow that UDP range between VMs or allow STUN/TURN if you rely on it. • Recommended network placement: same VPC/region or otherwise low‑latency network (high latency increases chance of timeouts). Keep firewall rules permissive for: ◦ TCP 443 (or 7880/7881 if not using TLS termination) outbound from workers to LiveKit ◦ UDP media ports between VMs (or configure TURN if NAT/firewall blocks UDP) • Summary: allow outbound WebSocket/TCP from worker → LiveKit, permit UDP media range between machines, and ensure low latency. 3) Recommended CPU / RAM sizing for your workload (10 concurrent SIP calls + Python voice agents) • Your current 4 vCPU / 7.5 GB RAM VM is undersized (CPU saturation caused the timeouts/401s). Recommended baseline: ◦ LiveKit server VM (recommended dedicated): ▪︎ Minimum: 4 vCPU (only if nothing else runs here and load is light) ▪︎ Recommended: 8 vCPU, 16 GB RAM (non‑burstable instance class) ▪︎ Reason: ensures headroom for signaling, participant init, routing and handling concurrent join spikes without timing out. ◦ SIP service VM (if separated): ▪︎ Recommended: 4–8 vCPU, 8–16 GB RAM depending on whether you perform transcoding or heavy media processing. ▪︎ If SIP does media transcoding or heavy DSP, lean toward the higher end (8 vCPU / 16 GB). ◦ Agent worker VM(s) (Python voice agent with OpenAI + Silero VAD): ▪︎ Per active agent process: ~4 vCPU, 8 GB RAM recommended. ▪︎ Options: • Run 1 agent per 4 vCPU / 8 GB VM (simpler, isolates CPU load), or • Run multiple agents on a larger VM (e.g., 8 vCPU / 16 GB for up to ~2 concurrent agents) if you monitor CPU headroom and keep per‑agent CPU usage low. ▪︎ Note: Python-based inference and audio processing can spike CPU; give headroom to avoid spikes blocking signaling on the LiveKit host. Practical deployment suggestions • Separate LiveKit server from agent workers (strongly recommended). Also strongly consider placing the SIP service on its own VM if it does media processing. • Use dedicated, non‑burstable instance types (avoid burstable/t‑class instances) so CPU performance is consistent. • Start by separating components this way, then run a controlled 10‑call load test and watch CPU, request latencies for /rtc/validate and signaling, and worker logs. • Aim to keep server CPU utilization comfortably below ~70–80% to avoid timeouts during spikes. Quick troubleshooting/check steps before/after separating: • After separation, re-run the 10 concurrent call test. If issues persist, collect: ◦ LiveKit server logs around the failed joins (10–20 lines before/after the timeout) ◦ Worker logs showing the “signal connection timed out” and any following 401s ◦ CPU/memory graphs for each VM during the test • If you want, send one representative log block and CPU snapshot and I’ll point to the most likely code path or bottleneck. If you’d like, I can: • Recommend specific instance types for your cloud provider (AWS, GCP, Azure) based on the sizes above. • Suggest a small test plan (staggered join script) to validate separation reduces failures. Sources: sip | livekit/livekit_composite | DeepWiki | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Self-hosted deployments | LiveKit Documentation | Running LiveKit on AWS