https://livekit.io logo
Join Slack
Powered by
Hi everyone We’re running a self-hosted multi-node...
# self-hosting
b
Hi everyone We’re running a self-hosted multi-node LiveKit deployment across two regions (Germany on-prem + Qatar on GCP). Both nodes run on microk8s with hostNetwork: true. Everything works great within the same node, but we’re hitting a consistent DTLS timeout when participants connect cross-region via TURN/TCP. Would really appreciate any insights from the community or the LiveKit team — especially from anyone running a similar on-prem multi-region setup with Kubernetes. Would really appreciate any insights from the community or the LiveKit team — especially from anyone running a similar on-prem multi-region setup. Setup • LiveKit v1.9.12, self-hosted, 2 nodes • Node 1: Germany (on-prem), Node 2: Qatar (GCP me-central1) • Shared Redis via WireGuard tunnel (~123ms latency) • TURN enabled with TLS passthrough via Contour/Envoy on port 443 • Both nodes registered in Redis, signaling works correctly Problem When a participant connects from the Gulf region to a room hosted on the Germany node: WSS signaling connects fine (proxied via Qatar → Germany) 1. ICE resolves to Germany TURN server via 
turns:
on TCP 443 Media works for ~10 seconds (audio + video both directions) 1. Then: 
dtls timeout: read/write timeout: context deadline exceeded
2. Video freezes, signaling stays connected Same-node connections work perfectly. Issue only occurs in cross-node scenarios. What we tried 
packet_buffer_size_video: 5000
, 
packet_buffer_size_audio: 2000
• OS UDP buffers increased to 5MB (
rmem_max
, 
wmem_max
) • Confirmed TCP 7881 connectivity between nodes • RTT ~130ms between client and TURN server TURN Config
Copy code
turn:
  enabled: true
  tls_port: 3478
  udp_port: 443
Q*uestions* 1. Is there a configurable DTLS timeout or keepalive interval for high-latency TURN/TCP scenarios? 2. We’re using Contour/Envoy as a reverse proxy for TLS termination (WSS) and TLS passthrough (TURN). Could this be causing the DTLS timeout? What reverse proxy setup do you recommend for self-hosted on-prem deployments? 3. For those running multi-node LiveKit on-prem — what does your production setup look like in terms of reverse proxy, TURN, and TLS? Any gotchas with high-latency cross-region TURN/TCP? 4. Any recommended configuration for self-hosted multi-node deployments with 100ms+ RTT between client and TURN?
f
Hi, check if UDP traffic is being allowed correctly, and the right address is being advertised during ICE
Whats the reason to spread the deployment over two separate regions?
what do you mean by 
WSS signaling connects fine (proxied via Qatar → Germany)
- is the connection to german region via a private network?
b
Copy code
UDP traffic: We've confirmed UDP is open on the GCP side (ports 50000-60000 and UDP 443). However, the on-prem Germany node is behind a FortiGate firewall — inbound UDP from the internet is blocked, which is why the client falls back to TURN/TCP. Within the same node, direct UDP works fine.

Reason for two regions: We serve users in both Europe and the Gulf region. Latency from the Gulf to Germany is ~130ms, so we wanted a closer node in Qatar for Gulf-based users to reduce latency for Gulf-only calls.

Signaling proxy: Yes, the two nodes are connected via a WireGuard VPN tunnel (private network). When a Gulf user connects to the Qatar node via WSS, but the room is hosted on the Germany node, LiveKit's built-in signaling bridge proxies the WebSocket messages from Qatar to Germany over this private network. The signaling part works — the participant appears in the room and we can see them briefly. The issue is specifically with the media/DTLS layer timing out after ~10 seconds.

So the flow is:

Gulf client → WSS → Qatar node (public) → signaling proxy via WireGuard → Germany node (room host)
Gulf client → TURN/TCP 443 → Germany TURN server → media (works ~10s then DTLS timeout)

The client is forced to use TURN/TCP to reach Germany because direct UDP is blocked by the firewall.
Update: We resolved the DTLS timeout issue The root cause was Contour/Envoy TLS passthrough interfering with the TURN/TCP connection. We replaced Envoy with HAProxy for SNI-based routing on port 443: - SNI 
<http://livekit-gulf.domain.com|livekit-gulf.domain.com>
→ HAProxy TLS termination → LiveKit WSS (7880) - SNI 
<http://livekitturn-gulf.domain.com|livekitturn-gulf.domain.com>
→ HAProxy TLS passthrough → LiveKit TURN (3478) - UDP 443 → LiveKit TURN UDP (direct, no proxy) Key: HAProxy with 86400s timeout + direct TLS passthrough for TURN fixed the issue. Envoy was apparently dropping the TURN/TCP connection after ~10 seconds. Cross-region now works: one participant from Germany (direct), one from Gulf via TURN/TCP through the Germany node — no freezing, stable audio/video. --- Follow-up question: In our current setup, all participants in a mixed-region room end up on the same node (the room's host node). The Gulf participant connects via TURN/TCP to the Germany node rather than having media served locally from the Qatar node. Is there any way in self-hosted LiveKit to achieve true distributed SFU — where each participant connects to their nearest node and media is relayed between nodes? Or is this exclusively a LiveKit Cloud feature? We'd love to keep self-hosting but this capability would significantly improve latency for our Gulf users in mixed-region calls.
f
Only cloud supports the distributed SFU architecture, self-hosted doesn't.
🙌 1
b
Thanks for confirming @flat-photographer-63184. So for self-hosted, here's our current architecture — would appreciate feedback on whether this is the recommended approach: Setup: - 2 nodes: Germany (on-prem, microk8s) + Qatar (GCP, microk8s) - Shared Redis via WireGuard VPN tunnel between nodes - Same API key/secret on both nodes - Redis Sentinel for HA (1 master Germany + 1 replica Qatar + 3 sentinels) Proxy layer (per node): - HAProxy on port 443 with SNI-based routing: - livekit.domain.com → TLS termination → LiveKit WSS (7880) - livekitturn.domain.com → TLS passthrough → LiveKit TURN (3478) - UDP 443 → LiveKit TURN UDP (direct, no proxy) How rooms work: - Gulf-only calls → room created on Qatar node (low latency for Gulf users) - Germany-only calls → room created on Germany node - Mixed calls (Gulf + Germany) → room on one node, remote participant joins via TURN/TCP through signaling proxy Planning to add: - Backend GeoIP routing via CF-IPCountry header to direct users to nearest node - regionaware node selector for automatic room placement Is this the right approach for multi-region self-hosted? Anything we should change or improve?
f
Yes, overall looks good to me, what do you mean by 
remote participant joins via TURN/TCP through signaling proxy
b
Thanks, to clarify: When a Gulf user joins a room hosted on the Germany node: 1. Gulf user connects WSS to Qatar node (livekit-gulf.domain.com) 2. Qatar node checks Redis, sees the room is on Germany node 3. Qatar node acts as a signaling bridge — proxies WebSocket messages to Germany node via internal network 4. For media (audio/video), the Gulf user's browser gets ICE candidates pointing to Germany's TURN server (livekitturn.domain.com:443) 5. Browser establishes TURNS/TCP connection to Germany's TURN server through HAProxy TLS passthrough 6. Media flows via this TURN relay to the Germany node where the room lives So the Qatar node only handles signaling proxy. Actual media goes directly from the Gulf client to the Germany TURN server via TURNS/TCP 443. This is why HAProxy with long timeouts was critical — Envoy was killing the TURN/TCP connection after ~10 seconds.
f
got it, yes thats the right flow
🎉 1
5 Views
Open in Slack
PreviousNext
Powered by