https://livekit.io logo
Join Slack
Powered by
how should be configured the digitalocean load bal...
# ask-ai
w
how should be configured the digitalocean load balancer with livekit server? We have this:
Copy code
# Ingress
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: livekit-mediaserver
  annotations:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: "cloudflare-issuer-for-livekit"
    <http://nginx.ingress.kubernetes.io/enable-cors|nginx.ingress.kubernetes.io/enable-cors>: "true"
    <http://nginx.ingress.kubernetes.io/cors-allow-origin|nginx.ingress.kubernetes.io/cors-allow-origin>: "*"
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "3600"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "3600"
    <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "3600"
    <http://nginx.ingress.kubernetes.io/proxy-buffering|nginx.ingress.kubernetes.io/proxy-buffering>: "off"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
      - <http://website.com|website.com>
    secretName: livekit-tls
  rules:
  - host: <http://website.com|website.com>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: livekit-server
            port:
              number: 443
In Digitalocean dashboard we have: • backend keepalive: disabled • http idle timeout 60 • proxy protocol: disabled Are we missing something? Wondering because we have edge cases with this errors: • Could not connect to room: could not establish signal connection: Failed to fetch • Could not connect to room: could not establish signal connection: Encountered websocket error during connection establishment • provider: connection aborted • Could not connect to room: could not establish pc connection
t
Thanks for reaching out, Alberto. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about how to configure the DigitalOcean Load Balancer for your LiveKit server and the connection errors you’re seeing. Short answer: Your current Ingress handles the HTTP/WebSocket signaling but won’t reliably handle WebRTC media (UDP). Two likely causes of your errors are (A) the load balancer is terminating or timing out long-lived WebSocket connections, and (B) UDP media (ICE) traffic is not being forwarded to LiveKit. Below I’ll explain what to check and concrete changes to try. What’s happening with your current setup • The Kubernetes Ingress you shared is an HTTP(S) (Layer 7) proxy. That will work for initial signaling (WebSocket over TCP/443) but it does not forward UDP traffic used by WebRTC. • DigitalOcean’s Application/HTTP load balancer (with an HTTP idle timeout = 60s) can close long-lived WebSocket connections unless keepalives/timeouts are aligned. • WebRTC requires UDP for peer connections (ICE/STUN/DTLS/SRTP). If UDP (LiveKit uses UDP port 7882) is not reachable, you’ll get “could not establish pc connection” and related media failures. What to check now • Confirm which DigitalOcean load balancer type you’re using: Application (HTTP(S)) vs Network (pass-through). If it’s an Application LB, it will not forward UDP. • Confirm whether UDP port 7882 is open on all firewalls (Droplet/K8s nodes) and forwarded by your load balancer. • Confirm whether the DO load balancer’s idle timeout (60s) may be closing WebSocket connections. Also check any NGINX ingress timeout overrides are actually applied (sometimes cloud LB overrides them). • Verify whether you rely on a TURN server. If clients can’t reach UDP, TURN over TCP can help but requires proper configuration and reachable ports. Concrete recommendations to fix / test 1. Use a Network (pass-through) load balancer or direct node ports for WebRTC UDP ◦ Switch to a DO Network/UDP-capable load balancer (or a passthrough) that forward both TCP and UDP to your nodes. ◦ Ensure the LB forwards: ▪︎ TCP 443 (signaling / WebSocket / HTTPS) ▪︎ TCP 7880 and TCP 7881 if you use those for internal services ▪︎ UDP 7882 (media) ◦ The Network LB should not terminate TLS for WebSocket/DTLS if you need client source IPs preserved. 1. Align timeouts and keepalives for WebSocket ◦ On the DigitalOcean LB: increase HTTP idle timeout to well above typical session durations (for example, 3600s to match your ingress config) if you're using an Application LB for signaling. Note: this alone does not solve UDP forwarding. ◦ On NGINX Ingress: keep your 
proxy-read-timeout
, 
proxy-send-timeout
, and 
proxy-connect-timeout
set high (you already have 3600s). ◦ Ensure proxy buffering is off (you already set 
proxy-buffering: "off"
). 1. Enable keepalives where appropriate ◦ If DigitalOcean LB supports backend keepalive, try enabling it to reduce connection churn. If enabling causes issues, try toggling it back — observe behavior. 1. Proxy protocol / TLS passthrough ◦ If you need original client IPs, enable proxy protocol where supported and ensure your backend supports it. ◦ If the LB terminates TLS (Application LB), make sure WebSocket TLS upgrade flows correctly to backend. Consider passthrough (Network LB) and terminate TLS at LiveKit if simpler. 1. TURN fallback ◦ Make sure you have a reachable TURN server (TURN over TCP/TLS) for clients behind restrictive networks. TURN over TCP/443 can help when UDP is blocked. ◦ Confirm LiveKit is configured with the TURN credentials and reachable host/ports. 1. Test connectivity with connection-check utilities ◦ Run WebSocket and UDP connectivity tests from the client network to the LB public IP to confirm: ▪︎ WebSocket (wss://) connects and remains open beyond 60s. ▪︎ UDP to port 7882 reaches the backend (or TURN) and STUN works. Optional Kubernetes considerations • If you switch to a Network LB that forwards UDP/TCP to nodes, the Ingress may become redundant for signaling. You can: ◦ Use the LB to forward 443 directly to the node/LiveKit Service 
NodePort
for signaling and UDP directly to the node. ◦ Or use a Service of type LoadBalancer that supports both TCP and UDP (if DO supports that) and expose the correct ports. Quick checklist you can run now • [ ] Is your DO load balancer a Network/pass-through LB? If not, consider switching. • [ ] Is UDP 7882 open and forwarded through the LB and node firewall? • [ ] Increase DO LB idle timeout (if you’re relying on Application LB signaling) to match ingress timeouts. • [ ] Ensure TURN is configured and reachable (TURN over TCP/443 as fallback). • [ ] Confirm NGINX ingress timeouts are actually in effect (no LB override). If you want, I can help further if you share: • Which type of DigitalOcean load balancer you’re using (Application vs Network). • Whether you have TURN configured and its ports/protocols. • How LiveKit is exposed in Kubernetes (Service type: ClusterIP/NodePort/LoadBalancer) and whether you’re using hostNetwork or a NodePort for UDP. I can then give step-by-step changes tailored to your cluster and DO LB type. Sources: Firewall Tips for LiveKit Connectivity | LiveKit | Diagnosing Connection Errors with Connection Test Utility | Running LiveKit on AWS | Running LiveKit on AWS
w
it seems we have an http load balancer instead of network load balancer, why was configured like this instead of network load balancer? Does it makes sense? How can we change it? If we change will websocket connection work?
t
Thanks — I’m responding to your three questions: why an HTTP LB was chosen, whether that makes sense for LiveKit, and how changing to a Network LB will affect WebSocket connections. Why an HTTP (Application) load balancer was likely chosen • Application LBs are the default for many Kubernetes Ingress setups because they terminate TLS, provide routing by hostname/path, and integrate with cert managers and Ingress controllers. If someone set up an NGINX Ingress and a DO Application LB in front of the cluster, they likely prioritized HTTPS termination, host/path routing, and automatic TLS issuance. • That choice is common for web apps where only TCP/HTTP(S) traffic is needed — it’s not uncommon to see this used by default even when the app also needs UDP. Does an HTTP LB make sense for LiveKit? • Short answer: it’s incomplete for LiveKit. • LiveKit uses WebSocket/TCP for signaling and UDP for media (ICE). An Application (HTTP) LB handles TCP/HTTPS and can proxy WebSocket signaling, but: ◦ The DO Application LB’s HTTP idle timeout (60s) can close long-lived WebSocket connections unless you increase it. ◦ It cannot forward UDP traffic (port 7882), so peer media connections can fail and you’ll see errors like “could not establish pc connection.” • So while the HTTP LB can work for signaling with correct timeout settings, it does not satisfy the UDP media requirement. For reliable LiveKit operation, a Network/pass-through LB (or another arrangement that forwards UDP) is required. If you change to a Network (pass-through) load balancer, will WebSocket connections work? • Yes — a Network/pass-through LB will forward both TCP and UDP without terminating the connection, which fixes two problems: ◦ UDP media (port 7882) will be forwarded to your nodes so ICE can establish media paths. ◦ TCP 443 traffic (WebSocket TLS) will be passed through; WebSocket upgrades will work as long as the backend (LiveKit/Ingress) handles TLS or you use TLS passthrough. • Caveats: ◦ If the Network LB simply passes TLS through to the backend, TLS termination must occur on the LiveKit server or on your ingress controller (i.e., you must have valid certificates there). ◦ If you still use an Application LB for TLS termination in front of the Network LB, you may re-introduce the UDP problem (avoid double-proxying). ◦ If you rely on host/path routing or cert automation at the LB layer, you’ll need to move that responsibility to the ingress controller or handle certs on the backend. How to change it (high level steps) 1. Decide where TLS is terminated: ◦ Option A: Terminate TLS at LiveKit/Ingress (use Network LB with TLS passthrough). Certificates must be configured on the ingress/LiveKit. ◦ Option B: Terminate TLS at an ingress controller (NGINX) and use a Network LB to forward TCP/UDP to nodes, with ingress handling host/path routing and certs. 1. Create/replace the DO load balancer as a Network/pass-through LB: ◦ In the DigitalOcean dashboard or using doctl/API, provision a Network load balancer that supports the needed protocols. ◦ Configure forwarding rules: ▪︎ TCP 443 -> your nodes (or NodePort for the ingress/backend) ▪︎ UDP 7882 -> your nodes (or NodePort for LiveKit) ▪︎ Add any other TCP ports you need (7880/7881) if they are in use ◦ Ensure health checks target appropriate TCP/HTTP endpoints and won’t falsely mark backends unhealthy. 1. Expose the LiveKit service on node ports (if needed): ◦ If DO’s Network LB forwards to node IPs+ports, create a Kubernetes Service of type NodePort for the ports that need LB forwarding (TCP 443 and UDP 7882), or configure hostNetwork where appropriate. ◦ Ensure node firewall rules allow those ports. 1. Open firewalls and security groups: ◦ Allow UDP 7882 and TCP 443 on nodes/Droplets and in any cloud firewall. 1. Update DNS if the LB IP changes: ◦ Point your domain to the new LB’s public IP. 1. Test: ◦ Test wss:// connection stability and that it remains open beyond previous timeout windows. ◦ Test ICE/STUN/TURN flow and confirm media flows without “pc connection” errors. Quick checks I recommend before changing • Confirm whether you currently have TURN configured as a fallback. TURN over TCP/443 can mitigate UDP-blocking clients until UDP is fully forwarded. • Tell me how your LiveKit is exposed in Kubernetes now (Service type: ClusterIP/NodePort/LoadBalancer? Are you using hostNetwork?). • Confirm whether you want to terminate TLS at the ingress or at LiveKit itself. If you share the current Service type and whether you rely on nginx ingress for TLS/cert management, I’ll give an exact step-by-step sequence you can apply in your cluster and in the DigitalOcean console to switch to a Network LB safely.
w
but this load balancer was created automatically in base of this:
Copy code
# Ingress
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: livekit-mediaserver
  annotations:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: "cloudflare-issuer-for-livekit"
    <http://nginx.ingress.kubernetes.io/enable-cors|nginx.ingress.kubernetes.io/enable-cors>: "true"
    <http://nginx.ingress.kubernetes.io/cors-allow-origin|nginx.ingress.kubernetes.io/cors-allow-origin>: "*"
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "3600"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "3600"
    <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "3600"
    <http://nginx.ingress.kubernetes.io/proxy-buffering|nginx.ingress.kubernetes.io/proxy-buffering>: "off"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
      - <http://website.com|website.com>
    secretName: livekit-tls
  rules:
  - host: <http://website.com|website.com>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: livekit-server
            port:
              number: 443
t
Thanks — I’m replying about why DO created an HTTP LB from your Ingress, whether that’s appropriate for LiveKit, and the next steps to switch to a Network LB safely. Why DigitalOcean created an HTTP load balancer • Your Ingress resource declares an NGINX ingressClass and TLS host routing. When you create an Ingress, many Kubernetes cloud integrations (including DigitalOcean’s) provision an Application/HTTP(S) Load Balancer automatically because the Ingress API is expressly for HTTP(S) routing, host/path rules, and TLS termination. • That behavior is expected: the cloud provider maps the Ingress intent (L7 routing + TLS) to an Application load balancer by default. Why the HTTP LB is incomplete for LiveKit • LiveKit requires both: ◦ TCP (WebSocket / HTTPS) for signaling (wss:// over 443), and ◦ UDP for media (ICE/STUN/DTLS/SRTP — LiveKit commonly uses UDP port 7882). • An Application (HTTP) LB handles TCP/HTTPS and can proxy WebSockets, but it cannot forward UDP traffic. Also, the DO Application LB has a default HTTP idle timeout (60s) which can close long-lived WebSocket connections unless increased. • Result: signaling may work inconsistently (timeout or proxy issues) and media (UDP) will fail — matching the errors you reported. Does switching to a Network (pass-through) LB make sense? • Yes — if you want reliable WebRTC media flow you need a Network LB (or another UDP-capable path) so UDP 7882 can reach your nodes. • Keep in mind switching affects where TLS is terminated and how routing/cert management are handled: ◦ If you use a Network LB, it typically passes TLS through to the backend (TLS termination moves to your ingress or LiveKit pod). ◦ If you relied on the Application LB for automatic certs/host routing, you’ll need to move that responsibility to your ingress (NGINX) or LiveKit. Will WebSocket connections still work after switching? • Yes. A Network LB will forward TCP 443 through unchanged, so WebSocket (wss://) will work as long as: ◦ TLS termination is correctly configured on the backend (NGINX or LiveKit) if you use passthrough; or ◦ If you keep TLS termination at the ingress, ensure the passthrough setup and certificates are in place. • Also ensure the LB doesn’t impose a small idle timeout; Network LBs generally do not terminate the connection so timeouts are much less likely to interfere. Practical next steps to change safely 1. Share current Service details ◦ Please paste the Kubernetes Service for your LiveKit server (type and ports). For example: is it ClusterIP, NodePort, LoadBalancer? Which ports are exposed and on what service ports? ◦ Also tell me whether LiveKit is using hostNetwork or a NodePort for UDP now. 1. Decide where TLS should terminate ◦ Option A: Terminate TLS at your ingress (NGINX). Use a Network LB that forwards TCP/443 and UDP/7882 to node ports; keep your Ingress certs and routing. ◦ Option B: Terminate TLS at LiveKit/ingress pod. Use Network LB with TLS passthrough and manage certs inside the cluster. 1. Create/replace the DO Load Balancer ◦ In the DO console or with doctl/API: create a Network (pass-through) load balancer and configure forwarding rules: ▪︎ TCP 443 -> node IPs / NodePort for your ingress/backend ▪︎ UDP 7882 -> node IPs / NodePort for LiveKit ◦ Ensure health checks are appropriate (TCP checks to an endpoint that validates the pod). 1. Expose node ports or use hostNetwork ◦ If DO’s Network LB forwards to node ports, expose the needed ports on a Service of type NodePort for TCP 443 and UDP 7882, or use hostNetwork if appropriate. 1. Open firewall rules ◦ Allow incoming TCP 443 and UDP 7882 on droplets/nodes. 1. Validate ◦ Test wss:// connections remain stable beyond previous timeouts. ◦ Test ICE (media) connectivity; validate TURN fallback if needed. Short-term mitigation (before switching) • Increase DO Application LB idle timeout (if you must keep it for a short period) to match your ingress timeouts (e.g., 3600s) to reduce WebSocket disconnects. • Ensure TURN is configured (TURN over TCP/443) so clients behind restrictive NATs can still connect while you migrate UDP forwarding. If you want, I can give an exact step-by-step manifest and DigitalOcean console/doctl commands once you share: • The Kubernetes Service manifest for livekit-server (type and ports). • Whether you want TLS terminated at ingress or at LiveKit. • Whether you have TURN configured (and its ports). Tell me those details and I’ll provide the concrete commands/manifests and the safe migration sequence.
w
it seems digitalocean has this annotation: metadata: annotations: service.beta.kubernetes.io/do-loadbalancer-type: "REGIONAL_NETWORK" Does it makes sense? Is there any downside?
t
Thanks — I’m responding to your question about the DigitalOcean annotation 
<http://service.beta.kubernetes.io/do-loadbalancer-type|service.beta.kubernetes.io/do-loadbalancer-type>: "REGIONAL_NETWORK"
, whether it makes sense for LiveKit, and any downsides. What this annotation does • Setting 
<http://service.beta.kubernetes.io/do-loadbalancer-type|service.beta.kubernetes.io/do-loadbalancer-type>: "REGIONAL_NETWORK"
tells DigitalOcean to provision a Network (pass-through) Load Balancer for that Kubernetes Service instead of an Application/HTTP(S) Load Balancer. • A Network LB forwards TCP and UDP traffic at layer 4 without terminating TLS or doing HTTP host/path routing. Why this makes sense for LiveKit • LiveKit requires UDP for media (commonly UDP 7882) in addition to TCP/HTTPS (signaling). A Network LB can forward UDP to your nodes, which an Application LB cannot. • Using a Network LB removes the Application LB’s HTTP idle timeout and TLS termination behavior that was causing WebSocket timeouts and blocked UDP media in your previous setup. • Overall, this annotation aligns the infrastructure with LiveKit’s requirements: signaling over TCP/443 and media over UDP. Potential downsides / considerations • TLS termination moves to your backend: ◦ The Network LB typically passes TLS through. You must terminate TLS on your ingress controller (NGINX) or directly in the LiveKit pods and manage certificates there (cert-manager + ingress is a common choice). • No L7 routing at the LB: ◦ Host/path routing and automatic certificate provisioning that an Application LB provides will not be handled at the LB layer. Keep your Ingress and cert-manager to continue hostname routing and certificate management. • Health checks and session handling: ◦ Choose appropriate health checks (TCP or custom) so the LB doesn’t mark nodes unhealthy. The checks are simpler at L4 versus L7, so plan accordingly. • NodePort / port exposure: ◦ The DO Network LB will forward to node IPs and node ports. You may need to expose the LiveKit ports (TCP 443 and UDP 7882) via a Service of type NodePort or use hostNetwork. Confirm your Service manifest. • Costs and regional behavior: ◦ Network Load Balancers may have different pricing or regional scope. Using 
REGIONAL_NETWORK
ensures regional availability but check DO billing/limits if cost is a concern. • Client IPs and proxy protocol: ◦ If you need original client IPs preserved, configure proxy protocol support and ensure your backend supports it. Without it, the backend will see the node/LB IPs. Short practical checklist to apply safely 1. Confirm your LiveKit Service exposes TCP 443 and UDP 7882 (NodePort or hostNetwork). 2. Add the annotation to the Service and apply it. This should create a Network LB that forwards both TCP and UDP. 3. Ensure certs and TLS are configured on the backend (NGINX or LiveKit), since LB won’t terminate TLS. 4. Open node/Droplet firewalls for TCP 443 and UDP 7882. 5. Configure health checks (TCP) that match a healthy LiveKit endpoint. 6. Test: verify stable wss:// signaling and successful ICE/media (no more “pc connection” errors). 7. Confirm TURN is configured as fallback for restrictive clients. If you’d like, paste the current Service manifest for 
livekit-server
(type, ports, any existing annotations). I’ll review it and provide exact manifest changes and the safe sequence to migrate to REGIONAL_NETWORK without downtime.
2 Views
Open in Slack
PreviousNext
Powered by