https://livekit.io logo
Join Slack
Powered by
"In the livekit design, If I am using haproxy load...
# ask-ai
b
"In the livekit design, If I am using haproxy loadbalancer beforw livekit node to manafer wss sigling and turn traffic via haproxy instead directly to livekit nodes, I have done all the setup and I deployed new Livekit setup in kubernetes , where Livekit is working with my existing deployed chat application call MMS service running os same cluster in different namespace, which is abstraction layer for managing the token, So architecture point of view I am using client (external) --------haproxy loadbalancer (wss or signaling sni media.example.com + WebRTC sni media-nturn .external.com connection on 3478 (UDP)) -------> istioingressGateway + Virtualservice, I have created two services into the istio-system for service one service called X4-ingressgateway which is recieving the all TCPrequest + other livekit-ingressGateway service for livekit traffic, Like wss(signaling) on port 443 TCP is getting managed by livekit-istioingressgateway -------> livekit-gateway which two endpoint one for livekit-quartz.external.com for (WSS request) and other route livekit-nturn-quartz.external.com (turn request on port 3478/UDP) connection with policy externalTrafficPolicy: Local or externalTrafficPolicy: Cluster. currently livekit cluster is running and integrated by chat service, also I am not using public IP directly NAT with livekit nodes, Istead I am reusing the Haproxy Public IP and configure Haproxy public IP in the livekit config.yaml in RTC section in my livekit version v1.9.4 use_external_ip: true also external_ip: "Public_IP_Of _Haproxy", - Currently I have deployed the livekit into the K8s as daemonset along with redis+Sentinel cluster. --------- Architecture ------------------ Model A – LiveKit Behind L4 LoadBalancer (Correct) HAProxy passes: TCP 443 → WSS Signaling UDP 3478 → LiveKit TURN (From haproxy --- Livekit node on 3478 direct via hostNetwork) UDP media ports 40k - 50k → (From haproxy --- Livekit node on 40k - 50k LiveKit advertises HAProxy public IP BUT HAProxy must be a pure L4 passthrough Issue: while initiating call getting two major issue: 1. Logs MMS service Room mimatch error. 2026/02/16 204248.541266 app.go997 [Warn] failed to load views: lstat /app/client/dist: no such file or directory {"file":"/app/pkg/models/notification_scheduler.go:49","func":"mms/pkg/models.(*NtfyScheduleModel).StartNtfyScheduler","level":"info","msg":"Ntfy duration at 5h20m11.458672017s","time":"2026-02-16T204248Z"} ┌───────────────────────────────────────────────────┐ │ media server version: 1.0.0 │ │ Fiber v2.52.9 │ │ http://127.0.0.1:8080 │ │ (bound on host 0.0.0.0 and port 8080) │ │ │ │ Handlers ........... 135 Processes ........... 1 │ │ Prefork ....... Disabled PID ................ 16 │ └───────────────────────────────────────────────────┘ 204258 | 401 | 19.879935ms | 127.0.0.6 | POST | /_mms/v2/auth/notification/room | - 204258 | 401 | 4.464213ms | 127.0.0.6 | POST | /_mms/auth/room/isRoomActive | - Engine initialization SUCCESS olm initialize using new context: memory manager = 0x7f6d1bc578b0, threading manager = 0x7f6d1bc57910, logging_provider = 0, crypto_systems = 0x7f6d1bc5cb20 Signature_valid is ::1 {"file":"/app/pkg/models/room_service.go:63","func":"mms/pkg/models.(*RoomService).LoadRoomInfo","level":"info","msg":"Requested room does not exist for the given roomId: !TArEBZroIjLSfNszvi:quartz1.service421.com","time":"2026-02-16T204258Z"} {"file":"/app/pkg/models/room_service.go:302","func":"mms/pkg/models.(*RoomService).LoadRoomWithMetadata","level":"info","msg":"load room info error from server for the given roomId: !TArEBZroIjLSfNszvi:quartz1.service421.com","time":"2026-02-16T204258Z"} {"file":"/app/pkg/models/room_auth.go:163","func":"mms/pkg/models.(*RoomAuthModel).IsRoomActive","level":"info","msg":"The room information: requested room does not exist","time":"2026-02-16T204258Z"} {"file":"/app/pkg/models/room_service.go:63","func":"mms/pkg/models.(*RoomService).LoadRoomInfo","level":"info","msg":"Requested room does not exist for the given roomId: !TArEBZroIjLSfNszvi:quartz1.service421.com","time":"2026-02-16T204258Z"} {"file":"/app/pkg/models/room_service.go:63","func":"mms/pkg/models.(*RoomService).LoadRoomInfo","level":"info","msg":"Requested room does not exist for the given roomId: !TArEBZroIjLSfNszvi:quartz1.service421.com","time":"2026-02-16T204258Z"} 204258 | 200 | 64.27182ms | 127.0.0.6 | POST | /_mms/auth/room/isRoomActive | - olm is already initialized: memory manager = 0x7f6d1bc578b0, threading manager = 0x7f6d1bc57910, logging_provider = 0, crypto_systems = 0x7f6d1bc5cb20 Signature_valid is ::1 {"file":"/app/pkg/models/room_service.go:63","func":"mms/pkg/models.(*RoomService).LoadRoomInfo","level":"info","msg":"Requested room does not exist for the given roomId: !TArEBZroIjLSfNszvi:quartz1.service421.com","time":"2026-02-16T204258Z"} {"file":"/app/pkg/models/room_service.go:302","func":"mms/pkg/models.(*RoomService).LoadRoomWithMetadata","level":"info","msg":"load room info error from server for the given roomId: !TArEBZroIjLSfNszvi:quartz1.service421.com","time":"2026-02-16T204258Z"} {"file":"/app/pkg/models/room_auth.go:163","func":"mms/pkg/models.(*RoomAuthModel).IsRoomActive","level":"info","msg":"The room information: requested room does not exist","time":"2026-02-16T204258Z"} ---------------- http://livekit.livekit.svc.cluster.local:7880 --- IynL2Bc7m6lChivgG3f1mZe7xAbtvp2avhq ------ cgXMxCM5TzmpVAqMLjkt63Uhdinp5VqKu 204258 | 200 | 80.986171ms | 127.0.0.6 | POST | /_mms/v2/auth/room/create | - olm is already initialized: memory manager = 0x7f6d1bc578b0, threading manager = 0x7f6d1bc57910, logging_provider = 0, crypto_systems = 0x7f6d1bc5cb20 2. Livekit is unable to advertize the ICE candidiates, I mean client is not getting livekit Server address along with dynamic port between 40k - 50k. --------- Architecture ------------------ Model A – LiveKit Behind L4 LoadBalancer (Correct) HAProxy passes: TCP 443 → LiveKit UDP 3478 → LiveKit TURN UDP media ports → LiveKit nodes LiveKit advertises HAProxy public IP BUT HAProxy must be a pure L4 passthrough Model B – Istio L7 + HAProxy (Problematic for WebRTC) Istio is great for: HTTP gRPC WSS But WebRTC media + TURN: Needs UDP Needs real client IP Needs stable port advertisement Needs symmetric NAT awareness Istio + HAProxy + externalTrafficPolicy misalignment = broken ICE negotiation. -------------- hapee config (enterprise) --------------------- # ============================================ # UDP PORT 3478 - STUN/TURN (UDP) - FIXED # ============================================ # REMOVE TPROXY/TRANSPARENT MODE - Use standard UDP load balancing # Option 1: Simple UDP configuration (recommended) listen udp_livekit_turn_simple mode udp bind :3478 # IMPORTANT: Use source IP hashing for session affinity balance source # Timeouts for WebRTC (long sessions) timeout client 4h timeout server 4h # Server options - use check-ssl for health checks if needed server lk01 lk01-quartz:3478 check server lk02 lk02-quartz:3478 check server lk03 lk03-quartz:3478 check # Option 2: Advanced UDP configuration with source port preservation frontend udp_livekit_turn mode udp bind :3478 # Use a backend that preserves source for better NAT traversal default_backend bk_livekit_turn_udp backend bk_livekit_turn_udp mode udp balance source # Force source IP preservation (critical for ICE) source 0.0.0.0 usesrc clientip # UDP specific settings option log-health-checks # Adjust these based on your network latency timeout check 5s timeout client 4h timeout server 4h # LiveKit TURN servers on UDP 3478 server lk01 lk01-quartz:3478 check server lk02 lk02-quartz:3478 check server lk03 lk03-quartz:3478 check # ============================================ # UDP PORTS 50000-60000 - TURN Relay Ports # ============================================ # This is for allocating relay candidates listen udp_livekit_relay mode udp bind :50000-60000 # Source affinity is critical here balance source timeout client 4h timeout server 4h # Each LiveKit node should have its own relay port range # Adjust port mapping based on your LiveKit configuration server lk01 lk01-quartz:50000-60000 check server lk02 lk02-quartz:50000-60000 check server lk03 lk03-quartz:50000-60000 check # ============================================ # HTTP Health Checks # ============================================ frontend livekit_health bind *:8080 mode http acl is_health path /health use_backend bk_health if is_health default_backend bk_default_http backend bk_health mode http option httpchk GET /livekit/health http-check expect status 200 server lk01 lk01-quartz:7881 check server lk02 lk02-quartz:7881 check server lk03 lk03-quartz:7881 check backend bk_default_http mode http server placeholder 127.0.0.1:80 disabled # ============================================ # Stats endpoint # ============================================ frontend stats bind *:1936 mode http stats enable stats uri /haproxy?stats stats refresh 10s stats show-legends no log livekit firewall rules opened in the Palo Alto --------------------------------------------------- Rule -1 Opend 3478/UDP & 40000 - 50000 UDP Internet ------------------------------------------> haproxy public IP Rule -2 opend 443/TCP 7880/TCP 5349/TCP Haproxy ------------------------------------------> livekit worker node Rule -3 opened 7880 & 7881 TCP livekit worker node ------------------------------- > livekit worker node Do I need to open the below rules also Rule -4 opend ? 40000 - 50000 UDP Haproxy ------------------------------------------> livekit worker node
✅ 1
t
Thanks for reaching out, devops. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a great, detailed description. I’ll respond to the two concrete problems you called out (room mismatch in MMS, and ICE/UDP candidate advertisement), confirm what must change in your network path, and answer your firewall question directly. Short summary of what I’m addressing • Problem 1: MMS shows “Requested room does not exist” even though a create call later returns 200. • Problem 2: Clients are not getting LiveKit ICE candidates / media port (40k–50k) — WebRTC fails to establish. • Your question: Do you need to open firewall rule for 40000–50000 UDP from HAProxy → LiveKit worker nodes? 1) Room mismatch in MMS (Requested room does not exist) • What that log means: LiveKit rooms are created on first successful join. If MMS calls IsRoomActive or otherwise expects the room to already exist before any client connects, it may see “room does not exist” until a participant actually joins. • Why you see a 200 for create afterwards: the MMS service or some other component is creating room metadata after the initial failed check or after the client attempts to join — this sequence can look like a mismatch. • What to do: ◦ Stop relying on pre-checks that require the LiveKit room to already exist before issuing a join token. Issue the token and allow the client to connect; LiveKit will create the room on first join. ◦ Use consistent room naming between MMS and LiveKit token generation (ensure the exact same roomId/roomName is used). ◦ Prefer reacting to LiveKit events (webhooks) for room lifecycle instead of polling or IsRoomActive checks that race with room creation. Have MMS consume room_started / participant_joined webhooks to update metadata/state reliably. 2) ICE candidates and UDP/media port advertisement (why candidates aren’t usable) • Core problem: UDP traffic (TURN 3478 and media ports 40k–50k) cannot traverse Istio L7 proxies reliably. Istio is suitable for HTTP / gRPC / WSS but not raw UDP media or TURN. • Consequences when UDP is sent through Istio (or mis-routed): ◦ Source client IPs and ports are not preserved → breaks TURN and NAT traversal. ◦ Relay ports advertised by LiveKit become unreachable or inconsistent. ◦ ICE negotiation fails because client can’t reach the advertised IP/ports. • Correct approach (Model A — requires UDP bypass of Istio): ◦ Allow WSS (signaling, TCP 443) to pass through HAProxy → Istio → LiveKit (this is okay). ◦ Ensure UDP 3478 (TURN) and UDP media port range (e.g., 40000–50000) are passed as L4 passthrough from HAProxy directly to LiveKit worker nodes (bypass Istio entirely). ◦ HAProxy for UDP must preserve client source IP (source affinity / usesrc clientip) so LiveKit sees real client IPs and can advertise the public IP/ports correctly. ◦ LiveKit nodes with hostNetwork and proper external_ip settings should then advertise the HAProxy public IP and correct port range. 3) HAProxy notes (what must be true) • HAProxy must operate as pure L4 for UDP (no TPROXY/transparency unless intentionally used and supported). Use source hashing/affinity and preserve source IP when possible: ◦ balance source ◦ usesrc clientip (if applicable to your HAProxy mode/environment) • For relay ports, map the same port range on HAProxy to the same port ranges on the nodes (so advertised ports remain stable). 4) Firewall question — do you need Rule 4 (40000–50000 UDP from HAProxy → LiveKit nodes)? • Yes. You must open UDP 40000–50000 (or whatever port_range_start/port_range_end your LiveKit config uses) and UDP 3478 from the HAProxy public IP to each LiveKit worker node. • Reason: HAProxy will receive UDP from the internet and then forward/route it to worker nodes. If HAProxy → nodes is blocked for those UDP ports, media/TURN traffic will be dropped and ICE will fail. • So your required rules are: ◦ Internet → HAProxy: UDP 3478 and UDP 40000–50000 (already open in your Rule-1 if that’s in place). ◦ HAProxy → LiveKit worker nodes: UDP 3478 and UDP 40000–50000 (this is Rule-4 you asked about — MUST be opened). ◦ Also keep HAProxy → LiveKit TCP ports open as you currently have (443/TCP for WSS, 7880/7881 for LiveKit control/health as you use them). 5) LiveKit config reminders (verify these values) • In LiveKit config.yaml: ◦ 
use_external_ip: true
◦ 
external_ip: "<Public_IP_of_HAProxy>"
◦ 
port_range_start
and 
port_range_end
must match the UDP range you expose on HAProxy and in firewall. ◦ TURN enabled and reachable on 3478 (if using LiveKit TURN relay). 6) Quick checklist for you to run through 1. Confirm LiveKit advertises external_ip equal to HAProxy public IP and that port_range matches firewall/HAProxy config. 2. Ensure HAProxy forwards UDP 3478 and UDP media ports as L4 directly to worker node IPs (not to Istio). 3. Open firewall Rule-4: HAProxy → LiveKit nodes for UDP 3478 and UDP 40000–50000. 4. Preserve client source IP in HAProxy UDP configuration (balance source / usesrc clientip where applicable). 5. Stop pre-checking room existence in MMS; use webhooks or allow join flow to create the room. 6. After changes, test with a client and verify the ICE candidates contain HAProxy public IP + a port in the advertised range and that STUN/TURN connectivity succeeds. If you want, I can: • Provide a concise HAProxy snippet for UDP (bind :3478 and bind :40000-50000) tuned to preserve source IP and session affinity. • Show the specific LiveKit config.yaml entries you should verify. • List the exact firewall rule lines you’ll need to add so you can hand them to your firewall team. Which of those would be most helpful now — HAProxy UDP snippet, LiveKit config snippet, or firewall rule lines? Sources: livekit/livekit_composite | DeepWiki | Firewall Tips for LiveKit Connectivity | LiveKit
b
"In the livekit design, If I am using haproxy loadbalancer beforw livekit node to manafer wss sigling and turn traffic via haproxy instead directly to livekit nodes, I have done all the setup and I deployed new Livekit setup in kubernetes , where Livekit is working with my existing deployed chat application call MMS service running os same cluster in different namespace, which is abstraction layer for managing the token, So architecture point of view I am using client (external) --------haproxy loadbalancer (wss or signaling sni media.example.com + WebRTC sni media-ntunr.external.com connection on 3478 (UDP)) -------> istioingressGateway + Virtualservice, I have created two services into the istio-system for service one service called X4-ingressgateway which is recieving the all TCPrequest + other livekit-ingressGateway service for livekit traffic, Like wss(signaling) on port 443 TCP is getting managed by livekit-istioingressgateway -------> livekit-gateway which two endpoint one for livekit-quartz.external.com for (WSS request) and other route livekit-nturn-quartz.external.com (turn request on port 3478/UDP) connection with policy externalTrafficPolicy: Local or externalTrafficPolicy: Cluster. currently livekit cluster is running and integrated by chat service, also I am not using public IP directly NAT with livekit nodes, Istead I am reusing the Haproxy Public IP and configure Haproxy public IP in the livekit config.yaml in RTC section in my livekit version v1.9.4 use_external_ip: true also external_ip: "Public_IP_Of _Haproxy", - Currently I have deployed the livekit into the K8s as daemonset along with redis+Sentinel cluster.
t
Thanks — this is clear and helpful. You’ve described your deployment and goals well. Before I give precise next steps and config snippets, I want to confirm what you’d like me to focus on so I don’t make assumptions. Quick recap of what you’ve described (so we’re aligned) • HAProxy public IP in front of your cluster; HAProxy handles external WSS and UDP (3478 + media ports). • Traffic path currently: Client → HAProxy → Istio IngressGateway + VirtualService → LiveKit services. • LiveKit runs as a DaemonSet with hostNetwork, Redis/Sentinel, and in config you set 
use_external_ip: true
/ 
external_ip: <HAProxy IP>
. • You have two Istio services / ingress objects: one handling TCP/WSS and another intended for TURN. You’ve considered externalTrafficPolicy settings. Before I give the specific recommended fixes (HAProxy UDP snippets, exact firewall rules, and LiveKit config checks), which of these do you want prioritized right now? Pick one or more: • A) Validate and provide a ready-to-use HAProxy configuration for UDP (TURN + media ports) that preserves client IP and session affinity. • B) Exact firewall rule list (source/dest/ports) you can hand to the network team to enable HAProxy → LiveKit node traffic. • C) LiveKit config.yaml snippet and checks (port_range_start/port_range_end, turn settings, external_ip verification). • D) Guidance on Istio usage: where it is okay (WSS) and where it must be bypassed (TURN/media), plus recommended routing pattern. • E) Troubleshooting checklist to diagnose room mismatch vs. ICE issues (what logs to inspect, tests to run, and how to verify candidates are advertised correctly). If you’re seeing specific errors (e.g., room-not-found logs or missing ICE candidates), tell me which one is highest priority and include a short recent log sample for that issue. I’ll then supply the targeted config and next-step commands. Which option would you like first?
b
rooms are not found … error only while making a call.
t
Thanks — I’m responding about the “rooms not found” error you see when making a call. Why this happens (short) • LiveKit creates rooms on the first successful participant join. If MMS calls IsRoomActive or otherwise expects the room to already exist before any client connects, LiveKit will report “room does not exist” until a participant with a valid token joins. • Race conditions occur when MMS checks room existence or waits for a CreateRoom response before allowing/issuing the join token. What to do (next steps — concise) 1. Stop pre-checking LiveKit room existence before issuing join tokens. Issue the token and let the client connect; LiveKit will auto-create the room on first join. 2. Verify the room name in the token exactly matches the room name MMS expects (the 
room
claim used when generating the token). 3. Use LiveKit webhooks (room_started, participant_joined) for MMS to update room state instead of polling or IsRoomActive checks. 4. If you need deterministic rooms, create room metadata in MMS before issuing the token but still allow LiveKit to create the room on first join (don’t gate the join on a prior LiveKit API check). 5. If you must call CreateRoom via API, be aware it can incur cross-region sync/latency — prefer the auto-create flow unless you have a strong reason. Quick checks to run now • Confirm the token your client uses contains the correct 
room
claim and valid expiry/signature. • Reproduce a join while tailing LiveKit logs to see a room_created / participant_joined event. • If a room is expected to be pre-created, check CreateRoom API timing and any errors returned. If you want, I can: • Provide a small webhook handling example for MMS to react to room_started. • Inspect one sample token (JWT) and a short recent log snippet to confirm token/room-name mismatch. Which would you like me to provide next? Sources: CreateRoom() is Taking Too Long | Avoiding Room ID Reuse When Re-Creating Rooms With the Same Name | Troubleshooting 401 Unauthorized and Token Errors | LiveKit, with interactive elements
3 Views
Open in Slack
PreviousNext
Powered by