Hi everyone, I’m running into a strange issue with a multi-node LiveKit deployment and could use some help debugging it. Setup: • Nodes:

lk-node1

,

lk-node2

,

lk-node3

,

lk-node4

(LiveKit), plus

lk-caddy

and

lk-redis

• Total: 4 LiveKit nodes • All nodes are connected to Redis over a private subnet • Public IPv4 is DNAT’d to private IPs (nodes cannot reach each other via public IP) Routing: •

lk-caddy

handles two domains: Main domain → routes to

lk-node1

TURN domain → load balances across all 4 nodes (`lk-node1`–`lk-node4`) Issues: 1. Node selection sometimes breaks —

lk-node1

fails to select other nodes, and this shows up on the client side as a signaling/WebSocket error (e.g., connection refused or repeated reconnects). 2. Participants occasionally join the same room but end up isolated (each on different nodes). Rejoining fixes it and places them correctly. 3. Noticeable slowness in WebSocket signaling when using multiple nodes (e.g., publishing new tracks)

Hey, I had a similar multi-node setup and ran into comparable issues. Not 100% sure but a few things worth checking: 1. Main domain only routes to node1 — this could be your biggest issue. Try load balancing across all 4 nodes. Any node can accept signaling and proxy to the room's host node via Redis. 2. Nodes can't reach each other via public IP — they need direct connectivity on TCP 7881 for signaling relay. Try setting node_ip to their private IP so inter-node communication works over the private subnet. 3. TURN load balancing across all nodes — this might cause problems. Each node has its own embedded TURN, and LiveKit returns the correct TURN server in ICE candidates automatically. Load balancing TURN externally could route clients to the wrong node. 4. Isolated participants — could be Redis latency or stale room state. Worth checking that all nodes are registered correctly (redis-cli HKEYS nodes) and Redis latency is low from all nodes. Hope this helps!

hello @brief-raincoat-81365 thank you for response

1. Main domain only routes to node1 — this could be your biggest issue. Try load balancing across all 4 nodes. Any node can accept signaling and proxy to the room's host node via Redis.

will modify it to be like that, I remember I faced some signalling issues with it

2. Nodes can't reach each other via public IP — they need direct connectivity on TCP 7881 for signaling relay. Try setting node_ip to their private IP so inter-node communication works over the private subnet.

this is tricky a little because when I set the node_ip to private then it will announce to participants on ICE which will make them cannot reach Livekit, how to work around it? currently node_ip point to NATTED public IP

3. TURN load balancing across all nodes — this might cause problems. Each node has its own embedded TURN, and LiveKit returns the correct TURN server in ICE candidates automatically. Load balancing TURN externally could route clients to the wrong node.

so it better to disable TLS TURN in all node except first one, will try it

4. Isolated participants — could be Redis latency or stale room state. Worth checking that all nodes are registered correctly (redis-cli HKEYS nodes) and Redis latency is low from all nodes.

will check it, for the latency I think we don't have any latency because redis is seperated VM in same subnet and all nodes connected to it using private IP

For #2 — try using both: rtc: node_ip: <PRIVATE_IP> use_external_ip: true node_ip handles inter-node communication, use_external_ip discovers public IP via STUN for client ICE candidates. Should solve the NAT issue. For #3 — I wouldn't disable TURN on other nodes. The problem is the external LB, not TURN itself. Remove the TURN load balancer and let LiveKit handle it — it advertises the correct TURN endpoint per client automatically. For #4 — since Redis is low latency, this is probably a side effect of #1. Fixing the LB should help.

> For #2 — try using both: > > rtc: > node_ip: PRIVATE_IP > use_external_ip: true > > node_ip handles inter-node communication, use_external_ip discovers public IP via STUN for client ICE candidates. Should solve the NAT issue. I cannot use use_external_ip because the node don't have public IP

Right, since your nodes only have private IPs with DNAT, use_external_ip won't work because STUN would return the NAT gateway IP, not the DNAT'd public IP. In that case, set node_ip to the public DNAT'd IP for each node:

rtc:
  node_ip: <PUBLIC_DNAT_IP>
  use_external_ip: false

This way clients get the correct public IP in ICE candidates. For inter-node communication, since all nodes are on the same private subnet with Redis, LiveKit uses Redis for signaling relay (TCP 7881) — it resolves the target node by the node_ip registered in Redis. So the trick is: make sure nodes can reach each other's node_ip on port 7881. If node_ip is set to public DNAT'd IP, then either: 1. Your NAT/firewall hairpins traffic (node1 public IP → routes back to node1 private IP internally) 2. Or add /etc/hosts entries on each node mapping the other nodes' public IPs to their private IPs Option 2 is simpler: # On node1 echo "<NODE2_PRIVATE_IP> <NODE2_PUBLIC_IP>" >> /etc/hosts This way when node1 tries to reach node2's public IP on 7881, it resolves to the private IP instead.

@brief-raincoat-81365 I will try second solution as first is impossible to do from my side