https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# ask-ai
s
This message was deleted.
t
Thanks for reaching out, karthi. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’ll address each of your three points for a private-VPC deployment where both the EC2 instance and the LB live in private subnets and clients connect via VPN/VPC. 1) Recommended load balancer (ALB vs NLB) • Use Network Load Balancer (NLB). LiveKit requires TCP/UDP passthrough for ICE/TURN and RTP candidate ports; ALB (Layer 7) cannot pass raw UDP or arbitrary TCP used by media/TURN. NLB supports the required TCP and UDP listeners and preserves client traffic for WebRTC/TURN. 2) Security group guidance (high level) • Keep a minimal, least-privilege model: allow only the required ports from the client network (your VPN/VPC CIDR) to the NLB, and allow only the NLB security group to reach the EC2 instance on the same ports. Permit SSH (or management) only from trusted admin IPs. 3) Concrete inbound rules and ports A. On the NLB (security group or equivalent network ACL entry from clients → NLB) • TCP 7880 — LiveKit signal (wss over TCP) — Source: client VPN/VPC CIDR • TCP 7881 — ICE/TCP fallback (if used) — Source: client VPN/VPC CIDR • UDP 3478 — TURN/UDP — Source: client VPN/VPC CIDR • TCP 5349 (or TLS port you choose) — TURN/TLS — Source: client VPN/VPC CIDR • UDP 50000–60000 (or your configured port range) — RTP/ICE media candidates — Source: client VPN/VPC CIDR B. On the LiveKit EC2 instance (security group allowing traffic only from the NLB SG) • Allow the same ports from the NLB security group: ◦ TCP 7880 ◦ TCP 7881 ◦ UDP 3478 ◦ TCP 5349 (or your chosen TLS port) ◦ UDP 50000–60000 (match LiveKit `port_range_start`/`port_range_end`) • Allow SSH (TCP 22) only from admin CIDR (not from clients) C. NLB -> EC2 listener/target configuration • Create TCP and UDP listeners on the NLB that map directly to the same ports on the EC2 target. • Forward: ◦ TCP 7880 → EC2 TCP 7880 ◦ TCP 7881 → EC2 TCP 7881 ◦ UDP 3478 → EC2 UDP 3478 ◦ TCP 5349 → EC2 TCP 5349 ◦ UDP 50000–60000 → EC2 UDP 50000–60000 • Use target groups with protocol passthrough (TCP/UDP). Enable preserve-client-ip if you need real client IPs in LiveKit logs/logic. Additional best practices and notes • Ensure LiveKit config matches these ports (signal port, rtc.tcp_port, port_range_start/end, turn.udp_port, turn.tls_port). • Use TLS for signaling (wss) and TURN/TLS for secure media. You can terminate TLS at a proxy or pass through to LiveKit/TURN; if you want NLB to handle TLS, bind listener on 443 and forward to your TURN TLS port. • Keep NLB and EC2 in the same VPC and subnets for low latency; use private IP routing from VPN to the NLB. • Restrict sources to your VPN/VPC CIDRs—do not open these ports to 0.0.0.0/0. • If running multiple LiveKit instances, use cross-zone NLB and unique port ranges per instance or use separate target groups. If you’d like, I can: • Provide an example LiveKit YAML snippet showing the port settings to match your SG/NLB rules • Draft Terraform or CloudFormation snippets for the NLB listeners, target groups, and security groups Which follow-up would you prefer? Sources: Deploy to Kubernetes | LiveKit docs | Deploying LiveKit | LiveKit docs | Ports and firewall | LiveKit docs | An open source stack for real-time multimodal AI
Good question — AWS listeners and target groups don’t accept a large explicit port range the same way a firewall does, so you need one of these approaches for LiveKit media ports (UDP 50000–60000): Options • Use NLB with listeners on the specific ports you need (not practical for 10k ports). • Use a single NLB UDP listener per port and many target groups — only feasible for a small number of ports. • Use NLB with the target group configured to use the instance’s port (target port = 0 / “traffic to instance port”) so the NLB preserves the client source port and forwards that UDP packet to the same port on the EC2 instance. This lets you open the full UDP range in the EC2 security group and on LiveKit without creating thousands of listeners. • Alternatively, run NLB in “preserve client port” passthrough mode (default for NLB UDP) and configure the NLB listener on the UDP port range you need by creating a listener on the UDP protocol and set the target group to forward to the instance (same port). Practically, you still create one UDP listener per needed external port — so prefer the preserve-client-port approach. Practical recommendation for LiveKit in your private VPC 1. Configure LiveKit to use a defined UDP port range (e.g., 50000–60000). 2. On the EC2 security group, allow UDP 50000–60000 from the NLB SG (or from client CIDR if NLB is internal). 3. On the NLB: ◦ Create UDP listeners for the ports clients will actually hit. To avoid creating thousands, have clients use the NAT/VPN route to the NLB and rely on NLB UDP passthrough that preserves the source port so packets to ephemeral media ports are forwarded to the same port on the instance. In practice this means a UDP listener that accepts traffic and forwards it to targets with “preserve client port” behavior. 1. Ensure target group references instance targets (use instance target type) so the instance receives packets on their original ports. 2. Verify LiveKit receives candidate RTP/ICE on the configured port range. If you want, tell me: • Are you using NLB with instance targets or IP targets? • Do you control the client port range or can you shrink it? I can then give the exact NLB listener/target-group setup steps or a concise Terraform example.
f
yes bes provide the setup steps, no need terraform code
t
Below are concise step-by-step setup steps to let an AWS NLB forward LiveKit’s UDP media port range (e.g., 50000–60000) to your EC2 instance without creating thousands of listeners. 1) LiveKit config • Set a fixed UDP port range in LiveKit config: ◦ 
rtc.port_range_start: 50000
◦ 
rtc.port_range_end: 60000
• Ensure 
turn.enabled: true
and set 
turn.udp_port
(e.g., 3478) and 
turn.tls_port
(e.g., 5349) as needed. 2) Choose target type • Use instance targets (target type = Instance). This allows NLB to forward traffic to the same destination port on the instance. 3) NLB listeners & target groups • Create listeners only for the protocols/ports you explicitly need: ◦ TCP 7880 → target group (TCP) → instance target (use instance port) ◦ TCP 7881 → target group (TCP) → instance target ◦ UDP 3478 → target group (UDP) → instance target ◦ TCP 5349 → target group (TCP) → instance target • For the UDP media range (50000–60000) do NOT create thousands of listeners. Instead: ◦ Create a single UDP listener on the NLB that will accept the media traffic. For NLB, you can create a UDP listener on one port and rely on “preserve client port” behavior when forwarding to instance targets. Practically, the usual approach is to create a UDP listener on the same ports clients will use; to cover a range, rely on VPN/VPC routing so clients send directly to NLB and the NLB preserves source/destination ports when forwarding to the instance. ◦ Ensure the NLB target group protocol is UDP and the target group forwards to the instance port (leave target port set to the instance port, not a fixed port mapping). • Key point: Use instance targets and UDP target groups so the packet’s destination port on the instance is the original client port (NLB preserves client source/destination port for UDP when using instance targets). 4) Security groups and routing • NLB is internal (private). Security group / NACL rules: ◦ NLB inbound (from client VPN/VPC CIDR): ▪︎ TCP 7880 ▪︎ TCP 7881 ▪︎ UDP 3478 ▪︎ TCP 5349 ▪︎ UDP 50000–60000 ◦ EC2 instance inbound (only allow from NLB SG): ▪︎ TCP 7880 (source = NLB SG) ▪︎ TCP 7881 (source = NLB SG) ▪︎ UDP 3478 (source = NLB SG) ▪︎ TCP 5349 (source = NLB SG) ▪︎ UDP 50000–60000 (source = NLB SG) • Do not open these to 0.0.0.0/0; restrict to your VPN/VPC CIDRs and NLB SG. • Allow SSH (TCP 22) only from admin CIDR. 5) Preservation of client port / IP • Use instance target type and UDP target groups so NLB preserves client ports; verify NLB target group settings forward to the instance port rather than remapping. • Verify LiveKit sees the client IPs if needed (NLB preserves source IP for TCP/UDP when configured with instance targets). 6) Verification steps • From a client inside your VPN, attempt a call and verify LiveKit logs show RTP/ICE candidates arriving on the UDP ports in the configured range. • Use tcpdump/ss on the EC2 instance to confirm UDP packets arrive on ports within 50000–60000. • Confirm TURN behavior by testing TURN allocation and relayed media. 7) Practical notes / alternatives • If preserving exact port mapping proves unreliable for your setup, reduce the port range (e.g., 50000–51000) to limit NLB complexity. • For TURN/TLS you can choose to have NLB handle TLS on 443 and forward to instance TLS port, or pass through TLS to LiveKit. • Keep NLB and EC2 in same AZs/subnets for best performance and enable cross-zone if you run multiple instances. If you want, I can now provide the exact sequence in the AWS console UI (which fields to choose when creating each UDP target group and listener) or a short checklist to follow in the console—which would you prefer?
f
Explain about more in this part guide me to configure step by step • For the UDP media range (50000–60000) do NOT create thousands of listeners. Instead: ◦ Create a single UDP listener on the NLB that will accept the media traffic. For NLB, you can create a UDP listener on one port and rely on “preserve client port” behavior when forwarding to instance targets. Practically, the usual approach is to create a UDP listener on the same ports clients will use; to cover a range, rely on VPN/VPC routing so clients send directly to NLB and the NLB preserves source/destination ports when forwarding to the instance. ◦ Ensure the NLB target group protocol is UDP and the target group forwards to the instance port (leave target port set to the instance port, not a fixed port mapping). • Key point: Use instance targets and UDP target groups so the packet’s destination port on the instance is the original client port (NLB preserves client source/destination port for UDP when using instance targets). remaining everything i done i struck only in part
t
Thanks — I'll give a clear, minimal step-by-step guide you can follow in the AWS console to ensure UDP media ports (50000–60000) are forwarded to your LiveKit EC2 instances without creating thousands of listeners. I’ll assume NLB is internal, your LiveKit instances are in the same VPC, and LiveKit is configured to use the 50000–60000 port range. Step 0 — prerequisites • LiveKit config on each instance: ◦ 
rtc.port_range_start: 50000
◦ 
rtc.port_range_end: 60000
◦ 
turn.enabled: true
and set 
turn.udp_port
if used (e.g., 3478) • EC2 security group allows UDP 50000–60000 (source: NLB SG or VPC CIDR) • Instances launched in the same subnets/AZs as the NLB Step 1 — create the NLB 1. EC2 Console → Load Balancers → Create Load Balancer → Network Load Balancer. 2. Name: e.g., 
livekit-nlb-media
. Scheme: Internal. IP type: IPv4. 3. Select the subnets/AZs where your LiveKit instances reside. 4. Create. Step 2 — create a UDP target group (instance targets) 1. EC2 Console → Target Groups → Create target group. 2. Target type: Instance. 3. Protocol: UDP. 4. Port: choose any valid port (e.g., 7882) — this is used for target-group definition/health-checks only. 5. VPC: choose your VPC. 6. Health check: Protocol TCP, Port 7880 (LiveKit signal port) so instances get verified via TCP. 7. Create the target group. 8. Register your LiveKit EC2 instances as targets (use the instance IDs). Save. Why instance targets: using instance targets ensures the NLB forwards UDP packets to the instance using the original destination port (port preservation). This is the key to avoid per-port listeners. Step 3 — add a UDP listener (one required by the console) 1. NLB → Listeners tab → Add listener: ◦ Protocol: UDP ◦ Port: pick an arbitrary port (e.g., 7882) — set Default action: Forward to the UDP target group you created. 1. Save. Note: The AWS console requires at least one listener. This listener satisfies that requirement. The critical behavior is that NLB will also accept and forward other UDP packets sent to the NLB IP on other ports (e.g., 50000–60000) to the instance targets with destination port preserved. Step 4 — security groups & routing 1. NLB is internal — ensure routing from your VPN/client CIDR to NLB IPs is in place. 2. NLB inbound (allow from client VPN/VPC CIDR): ◦ UDP 3478 (TURN/UDP) ◦ TCP 5349 (TURN/TLS) if used ◦ TCP 7880 (signal) ◦ TCP 7881 (ice/tcp) if used ◦ UDP 50000–60000 (media) 1. EC2 instance SG (allow only from NLB SG or VPC CIDR): ◦ UDP 50000–60000 — source = NLB SG (or VPC CIDR) ◦ UDP 3478 — source = NLB SG ◦ TCP 5349 — source = NLB SG ◦ TCP 7880, TCP 7881 — source = NLB SG ◦ TCP 22 — source = admin CIDR only Step 5 — ensure NLB preserves ports and forwards to instance port • Using instance targets + UDP target group means NLB will forward incoming UDP packets to the instance using the same destination port the client used (preserved). No per-port listener is required for each ephemeral port. • Verify the target group uses protocol UDP and target type Instance. Step 6 — LiveKit external IP / ICE candidate settings • Ensure LiveKit advertises the NLB IP/DNS for ICE candidates (via STUN or explicit external IP setting). This ensures clients get candidate addresses in the 50000–60000 range that map to the NLB IP. Step 7 — test & verify 1. From a client inside your VPN, start a WebRTC session and confirm media flows. 2. On the instance, run tcpdump/ss to confirm UDP packets arriving on ports in 50000–60000. 3. Check target group health (TCP 7880) and that instances are healthy. 4. If UDP packets do not appear, check VPC Flow Logs and SG/NACL rules. Troubleshooting notes • If your environment blocks arbitrary UDP ports between client and NLB (corporate VPN rules), reduce LiveKit’s port range (e.g., 50000–51000) to simplify debugging. • Ensure instance SG source equals the NLB SG (or VPC CIDR) so the NLB can forward packets. • If you need the NLB to terminate TLS for TURN/TLS, create separate TCP listeners for that (e.g., 443 → TCP 5349 target). If you want, I can now: • Walk you through the exact console clicks for creating the target group and listener with screenshots-style steps, or • Provide the exact health-check values and an example LiveKit config snippet to match these settings. Which follow-up would you prefer? Sources: Ports and firewall | LiveKit docs | Deploy to Kubernetes | LiveKit docs | Running LiveKit on AWS | Deploying LiveKit | LiveKit docs | Firewall Tips
Thanks — I reviewed your config and screenshots. You’re almost correct; the remaining gap is that NLB needs an appropriate UDP target group with instance targets so it can preserve destination ports for the 50000–60000 media range. Below are concise, actionable validation steps and fixes for your exact setup so UDP media reaches LiveKit without creating thousands of listeners. What I checked from your files/screenshots • LiveKit config already uses port_range_start: 50000 and port_range_end: 60000 — correct. • Docker compose uses host networking — correct and required. • You have NLB listeners and target groups for TCP 7880/7881 and TURN (TCP 5349, UDP 3478) — healthy. • EC2 SG shows rules allowing UDP 50000–60000 and other required ports from 10.0.0.0/8 — good, but tighten later to NLB SG. Gaps to fix (exact steps) 1) Ensure you have a UDP target group using target type = Instance • Console: EC2 → Target Groups → Create target group ◦ Target type: Instance ◦ Protocol: UDP ◦ Port: 1 (or any port — this value is not used for preserving client destination port; AWS requires a port) ◦ VPC: your VPC ◦ Health check: Protocol TCP, Port 7880 (LiveKit signal) — use TCP health check to validate instance • Register your LiveKit instance(s) as targets (use instance IDs). Save. Why: instance targets + UDP target group let NLB forward UDP packets to the instance with the same destination port (preserve client port). This is the core mechanism that avoids per-port listeners. 2) Ensure the NLB is associated with that UDP target group • Console: Load Balancers → select your NLB → Listeners ◦ Add or confirm a UDP listener that forwards to the UDP target group. ◦ If the console forces you to pick a single listener port, pick a small arbitrary port (e.g., UDP 7882) and forward to the UDP target group. Still, NLB will accept and forward UDP packets sent to NLB IP on any port to instance targets (port preserved) when target type is Instance. Why: the UI requires a listener entry, but NLB behavior preserves destination ports for UDP to instance targets even for ports without explicit listeners. 3) Security groups and source anchoring • NLB is internal — route clients (via VPN) to NLB IPs. • NLB SG inbound (source = your VPN CIDR): ◦ UDP 50000–60000 ◦ UDP 3478 ◦ TCP 5349 ◦ TCP 7880, TCP 7881 • EC2 instance SG inbound (source = NLB SG only): ◦ UDP 50000–60000 (From: NLB SG) ◦ UDP 3478 (From: NLB SG) ◦ TCP 5349 (From: NLB SG) ◦ TCP 7880, TCP 7881 (From: NLB SG) ◦ TCP 22 (admin CIDR only) Important: Don’t leave instance SG open to 10.0.0.0/8 long-term — use the NLB SG as the source. 4) LiveKit external IP / ICE candidate config • With NLB in front, LiveKit must advertise the NLB IP/DNS for ICE. Confirm one of: ◦ LiveKit can find external IP via STUN; or ◦ Set use_external_ip: true and ensure STUN is reachable (or set explicit external IP/DNS if needed). • Verify TURN domain in your config (you have turn domain entries and TLS configured in Caddy) maps to the NLB/TURN host. 5) Verify preservation & packet flow (tests) • From a client inside VPN: ◦ Attempt call; on the instance run: sudo tcpdump -n udp portrange 50000-60000 ◦ Confirm UDP packets arrive on ports in that range on the EC2 instance’s interface. • Verify target health: target group shows healthy (you already see healthy for 3478/5349/7880). • If packets don’t arrive, check: ◦ VPC Flow Logs for NLB IP and destination ports ◦ NACLs on subnets (must allow the UDP range) ◦ Route tables ensure VPN CIDR reaches NLB subnet 6) Common causes for agent timeout (what to inspect in logs) • ICE candidate not exchanged → check client logs and LiveKit logs for candidate discovery. • UDP blocked by client-side VPN — test UDP connectivity from client to NLB IP/port with nc/iperf. • LiveKit reporting different external IP than NLB — leads to clients trying wrong IP/port (fix via STUN or explicit external IP). • Instance SG still allows only 10.0.0.0/8 instead of NLB SG — tighten to NLB SG to ensure NLB actually forwards. Quick validation checklist you can run now 1. Confirm UDP target group exists with target type Instance and your EC2 instance registered. 2. Confirm NLB has a UDP listener forwarding to that UDP target group (UI may show one arbitrary port). 3. On EC2 run: sudo tcpdump -n -i any udp portrange 50000-60000 while a client attempts a call. 4. Check LiveKit logs for incoming ICE/TURN packets and any warnings about candidates or external IP. 5. Check VPC Flow Logs if tcpdump shows nothing. If tcpdump shows UDP arriving on instance ports but agent still times out: • Share the LiveKit server logs around the agent join attempt and a short sample of client-side logs — I’ll point to the exact mismatch (candidate, STUN, TURN or TLS handshake). If tcpdump shows no UDP arriving: • Share NLB listener and target group details (exact UDP target group settings and whether target type is Instance), and VPC Flow Logs for the attempt. I’ll identify whether it’s NLB routing, NACL, or VPN/route issue. Which test did you run already (tcpdump on EC2, LiveKit logs, or VPC Flow Logs)? Share one log snippet (LiveKit join error and local tcpdump output) and I’ll pinpoint the gap.
f
One correction In my EC2 SG rule, I whitelisted NLB SG id with all traffic port ( only one rule in availale in EC2 SG now) I attached that sg screeshot for NLB SG rules. here i have one doubt in below part, instead of 50000-60000 range port what port i need to use, please explain clearly 1) Ensure you have a UDP target group using target type = Instance • Console: EC2 → Target Groups → Create target group ◦ Target type: Instance ◦ Protocol: UDP ◦ Port: 1 (or any port — this value is not used for preserving client destination port; AWS requires a port) ◦ VPC: your VPC ◦ Health check: Protocol TCP, Port 7880 (LiveKit signal) — use TCP health check to validate instance • Register your LiveKit instance(s) as targets (use instance IDs). Save. Why: instance targets + UDP target group let NLB forward UDP packets to the instance with the same destination port (preserve client port). This is the core mechanism that avoids per-port listeners.
t
Thanks — good clarification. Short answer first, then exactly what to set and why. Short answer • For the UDP target group port you can pick any single valid port (common choices: 1, 7882, or 50000). The specific port value in the target group is not used to remap UDP destination ports when target type = Instance — NLB will preserve the original destination port the client used and forward that same port to the instance. So using port 1 or 7882 is fine; pick one you’ll recognize for health/debugging. What to set (step-by-step, minimal) 1. Target group • Target type: Instance • Protocol: UDP • Port: choose any (suggestion: 7882 or 1) — this only identifies the group. • VPC: your VPC • Health check: Protocol TCP, Port 7880 (LiveKit signal) 1. Register targets • Register your EC2 instance(s) by instance ID. 1. NLB listener • Add a UDP listener (required by console) on the same port you picked (e.g., UDP 7882) and forward to the UDP target group. Why this works (concise) • With instance targets and UDP, NLB preserves destination port and forwards packets to the instance using the same port the client sent. You do not need a separate listener for every media port in 50000–60000. The listener you create satisfies the console requirement; actual media packets to NLB_IP:5xxxxx will be forwarded to instance:5xxxxx. Security group notes (verify) • NLB SG: allow UDP 50000–60000 from your client/VPN CIDR. • EC2 SG: allow UDP 50000–60000 with source = NLB SG (you already added one "all traffic" rule; tighten to this specific range or keep as-is during testing). • Ensure NACLs and route tables allow the UDP range. How to verify quickly • On EC2 run: sudo tcpdump -n -i any udp portrange 50000-60000 • From client (over VPN) trigger a session; confirm packets arrive on those ports. • If packets arrive but agent still times out, share LiveKit join logs and a short tcpdump snippet; I’ll point to the mismatch (candidate, external IP, or TURN failure). If you want, I can recommend which single port to use for the target group in your setup (I suggest UDP 7882 to align with your other helper ports), and give the exact console click sequence. Which do you prefer?
f
I used 7882 port when i start call i getting IN and OUT packets by this command tcpdump -n -i any udp portrange 50000-60000 But livekit wss url is not reachable when i try to access it from my local machine FYI, i whitelisted 10.0.0.0/8 IP which our org internal network
t
Thanks — good progress that you see UDP media packets arrive. That means the media path is working. The remaining issue is signaling: WSS (TCP) to LiveKit is blocked or not routed. Here’s exactly what to check and fix next. 1) Confirm which WSS endpoint clients should use • If you want clients to connect to wss://livekitstaging.825382393.cloud.com (Caddy/TLS), they must reach the NLB or proxy that terminates TLS and forwards to LiveKit TCP 7880. • Decide: clients use port 443 (wss over standard TLS) or port 7880 (wss on custom port). I recommend 443 for simplicity. 2) Verify NLB/proxy listeners for signaling • NLB must have a TCP listener for the signaling path: ◦ If terminating TLS at Caddy on the instance: NLB TCP 443 → instance TCP 443 (Caddy) or TCP 7880 → instance 7880 if proxy forwards locally. ◦ If NLB passes through TLS to Caddy: TCP 443 listener forwarding to instance TCP 443. ◦ If using plain wss on 7880: TCP 7880 listener → instance TCP 7880. • Check your NLB listeners: you currently have TCP 7880 and 7881 shown — confirm TCP 443 exists if clients use standard wss. 3) Confirm Caddy is reachable and configured • Your caddy.yaml listens on :443 and proxies to localhost:7880 and 10.148.214.167:5349 per SNI. Ensure: ◦ Caddy is running (docker-compose host network gives it access to port 443). ◦ Certificates are valid for the domain clients use. ◦ Caddy accepts connections on the NLB IP/DNS (not only localhost). • Test from a client or jump host inside VPN: ◦ curl -vk https://livekitstaging.825382393.cloud.com/ (should return a response or TLS handshake) ◦ openssl s_client -connect <NLB_IP>:443 -servername livekitstaging.825382393.cloud.com 4) Security groups: ensure TCP for signaling is open to client CIDR • NLB SG: allow inbound TCP 443 (or TCP 7880 if using custom) from your org CIDR (10.0.0.0/8). • EC2 SG: allow inbound TCP 443 (or 7880) from NLB SG. • You mentioned EC2 SG allows all traffic from NLB SG — okay for testing. Ensure NACLs don’t block TCP 443/7880. 5) DNS resolution & where you test from • From your local machine, nslookup/resolve the domain and ensure it resolves to the NLB internal IP or reachable IP via VPN. • From inside VPN, run: telnet <domain> 443 (or 7880) or curl to confirm TCP connect. If DNS resolves to a public IP that’s not reachable via VPN, fix DNS to point to internal NLB. 6) Quick tests to run now (in order) • From a machine inside VPN: ◦ nslookup livekitstaging.825382393.cloud.com ◦ curl -vk https://livekitstaging.825382393.cloud.com/ ◦ telnet <resolved-ip> 443 (or nc -vz <ip> 7880) • On EC2 where LiveKit/Caddy runs: ◦ sudo ss -tunlp | grep 7880 ◦ docker-compose logs caddy | tail -n 100 ◦ sudo tcpdump -n -i any tcp port 443 or tcp port 7880 while attempting connection 7) If connections fail, common causes & fixes • DNS points to wrong IP: update to internal NLB IP or internal DNS. • NLB missing TCP listener for signaling: add TCP 443 (or 7880) listener forwarding to target group for instances. • TLS termination mismatch: if Caddy expects TLS, ensure listener is TCP 443 and traffic is TLS; if NLB terminates TLS, forward plain TCP to 7880. • NACL blocking TCP: allow TCP for signaling ports in subnet NACLs. 8) Next step I can do for you • If tcpdump shows UDP arriving (you confirmed) but curl/telnet to domain fails, share: ◦ Output of nslookup from the client ◦ Result of curl -vk (client) ◦ Output of sudo ss -tunlp on EC2 and docker-compose logs caddy ◦ NLB listeners screenshot showing TCP 443/7880 mapping I’ll pinpoint whether DNS, NLB listener, SG, or Caddy/TLS is blocking WSS. Want to send the curl/telnet outputs and ss/docker logs now? Sources: Ports and firewall | LiveKit docs | Running LiveKit on AWS | Deploy to a VM | LiveKit docs | Diagnosing Connection Errors with Connection Test Utility
f
443 port is not in my NLB listeners list, are you want me to add ?
t
Yes — add a TCP listener for signaling. Brief steps and checks: 1) Which listener to add • Preferred: TCP 443 (clients use standard wss:// on 443). Alternative: TCP 7880 (custom wss port) if you must. • I recommend TCP 443 so browsers/clients don’t need a port and TLS works normally. 2) Add listener & target group (console) • EC2 → Load Balancers → select your NLB → Listeners → Add listener ◦ Protocol: TCP ◦ Port: 443 ◦ Default action: Forward to a TCP target group (create one if needed) • Create target group if not present: ◦ Target type: Instance ◦ Protocol: TCP ◦ Port: 7880 (the LiveKit server port) ◦ Health check: TCP, Port 7880 ◦ Register your LiveKit EC2 instance(s) 3) Security groups & routing • NLB SG: allow inbound TCP 443 from your client/VPN CIDR (10.0.0.0/8). • EC2 SG: allow inbound TCP 443 (or 7880) from NLB SG. If Caddy listens on 443 on the instance, allow TCP 443; if NLB forwards to 7880, allow TCP 7880 from NLB SG. • Ensure subnet NACLs and route tables allow TCP 443. 4) TLS / Caddy mapping notes • If you forward TCP 443 → instance TCP 443, Caddy will terminate TLS (your caddy.yaml listens on :443). Good for SNI-based routing to LiveKit and TURN TLS. • If you forward TCP 443 → instance 7880, NLB is passing TLS through and Caddy must still terminate TLS on 7880 (ensure Caddy is configured accordingly). Simpler: forward 443→443. 5) Verify • From a client inside VPN: ◦ nslookup your domain -> should resolve to NLB IP ◦ curl -vk https://your-domain/ (expect TLS handshake/HTTP response) ◦ websocat or browser connect to wss://your-domain (should not timeout) • On EC2: ◦ sudo ss -tunlp | grep 443 or 7880 ◦ docker-compose logs caddy | tail -n 100 ◦ sudo tcpdump -n -i any tcp port 443 or tcp port 7880 while client connects 6) If it still fails, check these common problems • DNS resolves to an address not reachable via VPN (fix internal DNS to NLB IP). • EC2 Caddy not listening on the port you forwarded. • NACL blocking traffic. • Health checks failing (target group shows unhealthy -> no forward). If you want, I can list the exact console fields to click for creating the TCP 443 listener and the TCP target group, or review your curl/telnet output and ss/caddy logs once you add the listener. Which would you like?
f
Already i have configured 443 port with HTTPS in NLB SG, how again i can add 443 with TCP ?
t
Good question — you can’t have two listeners on the same port/protocol on one NLB. If you already have an HTTPS listener (TCP 443) configured at the NLB, you don’t add another TCP 443 listener. Instead confirm the existing 443 listener is forwarding to the correct TCP target group and instance port for signaling. Steps to validate/fix: 1. Inspect existing 443 listener • EC2 → Load Balancers → select your NLB → Listeners → click the 443 listener. • Check its Default action: which target group does it forward to? 1. Verify the target group for 443 • EC2 → Target Groups → open that target group. • Ensure: ◦ Target type = Instance ◦ Protocol = TCP ◦ Port = 443 (or Port = 7880 if you intend the NLB to forward 443 → 7880) ◦ Registered targets include your LiveKit EC2 instance(s) ◦ Health check: TCP on port 7880 (or 443 if Caddy listens on 443). Targets must be Healthy. 1. Match listener → instance behavior • If Caddy is terminating TLS on the instance (caddy.yaml listens on :443): ◦ Configure NLB 443 listener to forward to a TCP target group with target port = 443. • If you want NLB to accept 443 and forward plain TCP to LiveKit on 7880: ◦ Configure target group port = 7880 and forward 443 → that group. ◦ Ensure Caddy/LiveKit are listening on 7880 for proxied traffic (and TLS expectations are correct). 1. Security group checks (you said 443 allowed) • NLB SG: inbound TCP 443 from client CIDR (10.0.0.0/8) — OK. • EC2 SG: allow TCP 443 (if forwarding to 443) or TCP 7880 (if forwarding to 7880) from NLB SG — confirm this exists. 1. Quick verification steps • From a client inside VPN: ◦ nslookup livekit domain → confirm IP resolves to NLB internal IP ◦ curl -vk https://your-domain/ (should show TLS handshake) ◦ telnet <nlb-ip> 443 (or nc -vz <nlb-ip> 7880 if forwarding to 7880) • On EC2: ◦ sudo ss -tunlp | grep -E "443|7880" ◦ sudo tcpdump -n -i any tcp port 443 (attempt client connect) → verify packets arrive ◦ docker-compose logs caddy | tail -n 100 1. If the existing 443 listener forwards to an incorrect target group or port • Edit the listener → change the default action to forward to the correct target group (or create a correct TCP target group and point the listener to it). If you want, tell me: • Does the 443 listener currently forward to a target group with port 443 or 7880? I can then give the exact console clicks to change the target group or adjust the target group's port/health check so WSS will reach your LiveKit.
f
currently from NLB , TCP 443 listener forward traffic to a target group with port 7880 find the current NLB listeners configuration screenshot FYI, in NLB SG, i have configured HTTPS 443 with my internal network IP (10.0.0.0/8) I want to know why livekit domain is not accessible
t
Thanks — useful detail. You have UDP working but signaling (WSS) still failing because the 443 listener forwards to port 7880. That can break TLS/WSS unless Caddy/LiveKit are listening/terminating TLS on the port the NLB forwards to. Here’s exactly how to fix and verify so your domain becomes reachable. Options — pick one and implement Option A — Terminate TLS on the instance (recommended with your Caddy) 1. Configure NLB 443 listener to forward to target group whose target port = 443 (instance must have Caddy listening on 443). 2. Ensure Caddy is running and has the certificates for livekitstaging.* and is listening on :443 (your caddy.yaml already uses :443). 3. EC2 SG: allow inbound TCP 443 from NLB SG. NLB SG: allow inbound TCP 443 from client CIDR (10.0.0.0/8). 4. Health check: set target-group health check to TCP 443 (or keep TCP 7880 if you prefer but ensure health path matches). 5. Verify: ◦ From a VPN client: curl -vk https://livekitstaging.825382393.cloud.com/ (should show TLS handshake). ◦ Browser: open wss://livekitstaging.825382393.cloud.com (should connect). ◦ On instance: sudo ss -tunlp | grep :443 and docker-compose logs caddy. Option B — Let NLB accept 443 and forward TLS to Caddy on 7880 (less common) 1. Keep NLB 443 → target group port = 7880. 2. Reconfigure Caddy to accept TLS on port 7880 (not typical) or ensure it will accept proxied TLS on 7880. 3. EC2 SG: allow inbound TCP 7880 from NLB SG. 4. Health check: TCP 7880. 5. Verify as above. (This approach is unusual — prefer Option A.) Option C — Use NLB TCP 7880 (non-standard port) 1. Have clients connect to wss://domain:7880 (requires client change). 2. NLB TCP 7880 → target group port 7880; EC2 SG allow 7880 from NLB SG. 3. Verify connectivity. (Only use if you cannot use 443.) Immediate checks for your current state 1. Is Caddy actually listening on port 443 inside the instance? ◦ Run: sudo ss -tunlp | grep :443 ◦ If no process, Caddy isn’t terminating TLS — Option A will fail until Caddy listens on 443. 1. From a VPN client: ◦ nslookup livekitstaging.825382393.cloud.com → resolves to NLB IP? ◦ curl -vk https://livekitstaging.825382393.cloud.com/ → observe handshake or error. ◦ openssl s_client -connect <nlb-ip>:443 -servername livekitstaging.825382393.cloud.com 1. On EC2 while you attempt connect: ◦ sudo tcpdump -n -i any tcp port 443 or tcp port 7880 — do packets arrive? 1. Target group health: ◦ Ensure target group used by 443 listener shows targets Healthy for the same port you expect Caddy to be on. Most likely cause in your setup • NLB 443 → target group port 7880 but Caddy is configured to terminate TLS on 443. NLB is forwarding TLS to port 7880 where no TLS listener exists → TLS handshake fails and WSS won’t connect. Either make NLB forward to instance port 443 (Option A) or make Caddy listen/terminate TLS on 7880 (Option B). If you want, I’ll: • Tell you the exact console clicks to change the 443 listener’s target group port to 443, or • Walk through the commands to verify Caddy is listening and show the tcpdump/curl outputs to interpret errors. Which do you want me to do next?
f
I think caddy not providing certificate. {"level":"info","ts":1765876421.8184078,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00079a280"} {"level":"info","ts":1765876421.8197017,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"} {"level":"info","ts":1765876421.8197217,"msg":"serving initial configuration"} {"level":"info","ts":1765876421.8238215,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data","instance":"75f68166-2573-4255-97f2-e3858f14d1cc","try_again":1765962821.8238182,"try_again_in":86399.999998931} {"level":"info","ts":1765876421.8238912,"logger":"tls","msg":"finished cleaning storage units"} {"level":"error","ts":1765876423.0911083,"logger":"layer4","msg":"matching connection","remote":"10.148.214.77:14766","error":"EOF"} {"level":"error","ts":1765876424.8418882,"logger":"layer4","msg":"matching connection","remote":"10.148.214.12:59628","error":"EOF"} {"level":"error","ts":1765876433.0906205,"logger":"layer4","msg":"matching connection","remote":"10.148.214.77:53850","error":"EOF"} {"level":"error","ts":1765876434.8421388,"logger":"layer4","msg":"matching connection","remote":"10.148.214.12:9931","error":"EOF"} {"level":"error","ts":1765876443.0909598,"logger":"layer4","msg":"matching connection","remote":"10.148.214.77:22503","error":"EOF"} FYI, when do curl with localhost:7880 inside ec2 i am getting response, with domain only not accessible
t
Thanks — the Caddy logs confirm it’s not obtaining TLS (and your earlier EOFs mean TLS handshakes are being dropped). Here’s a concise action plan to get WSS working (do these in order and report back results or any failing step). 1) Ensure DNS resolves to the NLB • From a machine inside your VPN run: ◦ dig livekitstaging.825382393.cloud.com +short • Result must be the NLB DNS/IP. If not, update DNS to point to the NLB. 2) Enable HTTP (port 80) so Caddy can complete ACME HTTP-01 • Add NLB listener: TCP 80 -> target group -> instance port 80 (create TG if needed). • NLB SG: allow inbound TCP 80 from your client/VPN CIDR (10.0.0.0/8). • EC2 SG: allow inbound TCP 80 from NLB SG. • Verify with: curl -v http://livekitstaging.825382393.cloud.com/ from inside VPN (should not time out). 3) Confirm 443 listener forwards to instance port 443 (recommended) • In NLB, ensure TCP 443 listener forwards to a TCP target group where target port = 443 (not 7880). • EC2: ensure Caddy is listening on 443 (run: sudo ss -tunlp | grep :443). • EC2 SG: allow inbound TCP 443 from NLB SG. 4) Start/verify Caddy certificate provisioning • Restart Caddy (docker-compose restart caddy). • Watch logs: docker-compose logs -f caddy • You should see: "certificate obtained successfully" for your domain. • If errors appear, paste the relevant log lines. 5) Verify WSS / signaling • From inside VPN: ◦ curl -vk https://livekitstaging.825382393.cloud.com/ (TLS handshake) ◦ wscat -c wss://livekitstaging.825382393.cloud.com/rtc (or open browser client) • On EC2 while testing run: ◦ sudo tcpdump -n -i any tcp port 80 or tcp port 443 ◦ docker-compose logs livekit | tail -n 100 6) If NLB forwards 443 → 7880 (current state) and you prefer keeping that: • Ensure Caddy is configured to terminate TLS on 7880 (not typical). Easier: change NLB to forward to 443 and let Caddy handle TLS. 7) Quick checks if cert provisioning still fails • DNS A record must point to NLB. • Port 80 traffic from Let’s Encrypt must reach Caddy (if private VPC, ensure Caddy can reach ACME servers outbound). • If your environment blocks public ACME, use a certificate from your CA and configure Caddy to use it (or use DNS-01 challenge via a supported provider). 8) Next diagnostics if WSS still unreachable after above • Share outputs: ◦ dig +short <domain> ◦ curl -vk https://<domain> (client side) ◦ sudo ss -tunlp | grep -E "443|7880" (server) ◦ docker-compose logs caddy (relevant error lines) ◦ tcpdump output for tcp port 443 during a connect attempt You already have media UDP working (tcpdump showed packets). Focus is to get TLS working so signaling (WSS) succeeds. Which step would you like to try now? Sources: Ports and firewall | LiveKit docs | Deploy to a VM | LiveKit docs | Deploying LiveKit | LiveKit docs | Configuring firewalls | LiveKit docs | Deploy to Kubernetes | LiveKit docs
f
as per your recommendation, I add 80 port listner with 7880 health check, target port also 80 {"level":"info","ts":1765881307.1106348,"msg":"using config from file","file":"/etc/caddy.yaml"} {"level":"info","ts":1765881307.1114776,"msg":"adapted config to JSON","adapter":"yaml"} {"level":"info","ts":1765881307.1132538,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[:1]2019"]} {"level":"info","ts":1765881307.116894,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000030780"} {"level":"info","ts":1765881307.1228688,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"} {"level":"info","ts":1765881307.1228948,"msg":"serving initial configuration"} {"level":"info","ts":1765881307.1262162,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data","instance":"75f68166-2573-4255-97f2-e3858f14d1cc","try_again":1765967707.1262124,"try_again_in":86399.999998979} {"level":"info","ts":1765881307.1268663,"logger":"tls","msg":"finished cleaning storage units"} {"level":"error","ts":1765881313.0901787,"logger":"layer4","msg":"matching connection","remote":"10.148.214.77:64259","error":"EOF"} {"level":"error","ts":1765881314.8418467,"logger":"layer4","msg":"matching connection","remote":"10.148.214.12:41785","error":"EOF"} due to this domain url not working? FYI, when do curl with localhost:7880 inside ec2 i am getting response, with domain only not accessible (edited)
t
Good — thanks for the logs. You added a port 80 listener, but Caddy still shows EOF on incoming connections and the domain remains unreachable. That means HTTP/ACME challenges or TLS handshakes still aren’t completing. Do the following checks in order and report results for the failed step you want help with. 1) DNS → NLB • From a VPN client run: ◦ dig livekitstaging.825382393.cloud.com +short • Expect the NLB’s IP/DNS. If it does not resolve to the NLB, update DNS. 2) Confirm NLB listeners & target groups • Listeners: TCP 80 -> TG(port 80), TCP 443 -> TG(port 443). • Target groups: TG(port 80) and TG(port 443) must be Instance type and include your EC2. • Health checks: TG for 80/443 use TCP health check on same port (or 7880 if you want to healthcheck LiveKit directly). 3) Verify security groups and NACLs • NLB SG: inbound allow TCP 80 and TCP 443 from client/VPN CIDR (10.0.0.0/8). • EC2 SG: inbound allow TCP 80 and TCP 443 from NLB SG. • Subnet NACLs allow 80/443 traffic both directions. 4) Confirm Caddy listening and cert attempt • On EC2: ◦ sudo ss -tunlp | grep -E '80|443' ◦ docker-compose logs -f caddy (watch while you attempt a curl from VPN) • Expected: Caddy listening on 0.0.0.0:80 and 0.0.0.0:443 and logs showing ACME challenge activity and eventually “certificate obtained successfully”. 5) Test from a VPN client • curl -vk http://livekitstaging.825382393.cloud.com/ ◦ If this times out: NLB/DNS/SG/NACL issue. ◦ If this returns but cert still missing: Caddy ACME outbound blocked. • curl -vk https://livekitstaging.825382393.cloud.com/ (after cert present should handshake) 6) Check Caddy ACME outbound access • Caddy must be able to reach Let's Encrypt (outbound). From EC2: ◦ curl -v https://acme-v02.api.letsencrypt.org/directory ◦ If this fails, allow outbound internet for the instance (NAT gateway or allow outbound via existing route). 7) Inspect tcpdump while testing • On EC2 run while you curl from VPN: ◦ sudo tcpdump -n -i any tcp port 80 or tcp port 443 • If packets reach EC2 but Caddy logs EOF, paste the tcpdump lines and the matching caddy log errors. Common root causes given your symptoms • DNS not pointed at NLB (curl to domain times out). • NLB listener exists but forwards to port 7880 where Caddy is not listening for TLS (you should forward to instance port 443 if Caddy terminates TLS). • Caddy cannot complete ACME because instance has no outbound internet (NAT) or port 80 not reachable by Let’s Encrypt. • SG/NACL blocking 80/443 between NLB and instance. Tell me the outputs for any one of these and I’ll pinpoint the exact change: • dig +short <domain> • curl -vk http://<domain> • sudo ss -tunlp | grep -E '80|443' • docker-compose logs caddy (last 50 lines) • tcpdump output while you curl (a few lines) Which one would you like to share first?
f
1. I checked with this command dig livekitstaging.825382393.cloud.com +short from my local, i receiverd two IP's 2. Listeners pointed: TCP 80 -> TG(port 80), TCP 443 -> TG(port 443). Target groups: TG(port 80) and TG(port 443) is Instance type and include my EC2. Health checks: TG for 80/443 use TCP health check on 7880 3. what is NACLs and in NLB SG i allowed TCP 80 and TCP 443 from client/VPN CIDR (10.0.0.0/8). in EC2 SG: inbound i whitelisted NLB SG id with all traffic, Hence no need to add separtely right? I allowed all traffic with 0.0.0.0 in Subnet NACLs so need to add seprately that 80 and 443 right? 4. i checke with this command sudo ss -tunlp | grep -E '80|443' and i got below output tcp LISTEN 0 4096 *:443 : users:(("caddy",pid=12839,fd=7)) 5. I tested from my local with this copmmand curl -vk http://livekitstaging.825382393.cloud.com/ i Got below output HTTP/1.1 502 badgateway 6. i Checked Caddy ACME outbound access, it is accessible Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 7. i checked with this sudo tcpdump -n -i any tcp port 80 or tcp port 443 command in ec2 I am getting packets contineously few things FYR 105526.977705 enX0 In IP 3.78.203.51.443 > 10.148.214.14.42286: Flags [.], ack 71266, win 443, options [nop,nop,TS val 1059855618 ecr 1701168123], length 0 105527.034224 enX0 In IP 3.78.203.51.443 > 10.148.214.14.42286: Flags [P.], seq 27228:27552, ack 71266, win 443, options [nop,nop,TS val 1059855674 ecr1701168123], length 324 105527.034373 enX0 Out IP 10.148.214.14.42286 > 3.78.203.51.443: Flags [.], ack 27552, win 2170, options [nop,nop,TS val 1701168182 ecr 1059855674], length 0 Please find my outputs
t
Thanks — useful outputs. You already have UDP media arriving and Caddy listening on 443. The 502 Bad Gateway plus Caddy EOF logs indicate HTTP (port 80) requests reach the instance but Caddy isn’t answering HTTP properly for ACME challenges or proxying to LiveKit. Do these exact steps to fix and verify: 1) Make Caddy listen on port 80 • Edit your Caddy config (caddy.yaml) to ensure there is a listener for :80 (HTTP) and an ACME challenge route. Your current config shows only :443. Add/confirm a section like: ◦ listen: [":80"] ◦ ensure HTTP -> HTTPS or ACME challenge handling exists for your domains. • Restart Caddy: docker-compose restart caddy 2) Confirm Caddy actively listens on 80 and 443 • Run on EC2: sudo ss -tunlp | grep -E '80|443' • Expect processes bound to both :80 and :443 (caddy). 3) Verify NLB → Instance mapping and health checks • NLB listener TCP 80 → target group with port 80 (Instance targets). • Target group health check: TCP on port 80 (so health matches actual listener). • NLB listener TCP 443 → target group port 443 (instance Caddy TLS). • Ensure both TGs show the instance Healthy. 4) Test HTTP reachability from client/VPN • From your client: curl -vk http://livekitstaging.825382393.cloud.com/ • If you get a 200/302/301 or HTTP response (not timeout or immediate 502), ACME HTTP-01 will work. • While testing, watch Caddy logs: docker-compose logs -f caddy 5) If curl to HTTP still returns 502 • Capture tcpdump during the curl and paste the few lines around the request: sudo tcpdump -n -i any tcp port 80 or tcp port 443 -c 50 • Paste the last 10–20 lines of Caddy logs after running curl. • A 502 from NLB often means Caddy responded with an error (bad upstream) or the target group health check is mismatched (TG expects 7880 but listener is 80). Ensure the TG for port 80 points at instance port 80. 6) Confirm ACME outbound works (you already did) • You showed Let’s Encrypt reachable — good. Once Caddy serves HTTP on :80 and DNS points to NLB, Caddy should obtain certs and logs will show “certificate obtained successfully”. 7) Quick checklist (tick off) • DNS A/alias → NLB (you said dig returns IPs) ✅ • NLB listener 80 → TG(port 80) and TG healthy ✅/verify • EC2 SG allows 80/443 from NLB SG ✅ • Caddy listening on 80 and 443 — verify (step 2) • curl http succeeds (step 4) If you still see 502 after making Caddy listen on 80 and reloading: • Share the last 20 lines of: docker-compose logs caddy • Share the tcpdump capture from step 5 I’ll pinpoint whether it’s upstream (proxy to 7880), TG health mismatch, or a Caddy routing error.
f
This my caddy yaml please change and provide as per your suggestion logging: logs: default: level: INFO storage: "module": "file_system" "root": "/data" apps: tls: certificates: automate: - livekitstaging.825382393.cloud.com - livekitstaging-turn.825382393.cloud.com layer4: servers: main: listen: [":443"] routes: - match: - tls: sni: - "livekitstaging-turn.825382393.cloud.com" handle: - handler: tls - handler: proxy upstreams: - dial: ["10.148.214.166:5349"] - match: - tls: sni: - "livekitstaging.825382393.cloud.com" handle: - handler: tls connection_policies: - alpn: ["http/1.1"] - handler: proxy upstreams: - dial: ["localhost:7880"]
t
Thanks — I updated your Caddy config above. Next steps to apply it and get WSS working (do these in order): 1) Deploy the new caddy.yaml • Replace your current file with the YAML I provided. • Create ACME challenge directory and set permissions: sudo mkdir -p /data/acme-challenges sudo chown -R 1000:1000 /data • Restart Caddy: docker-compose restart caddy 2) Confirm Caddy is listening on 80 and 443 • Run on the EC2 host: sudo ss -tunlp | grep -E '80|443' • Expect caddy bound to both :80 and :443. 3) Verify HTTP reachability (from a client inside your VPN) • From your client: dig livekitstaging.825382393.cloud.com +short curl -vk http://livekitstaging.825382393.cloud.com/ • Expected: HTTP request reaches Caddy (should not time out). If you get 502, capture Caddy logs and tcpdump per step 5. 4) Watch Caddy logs for ACME activity • On EC2: docker-compose logs -f caddy • Look for lines like: "certificate obtained successfully" for both domains. 5) If HTTP returns 502 or TLS not issued, collect these and share: • Output of: sudo tcpdump -n -i any tcp port 80 or tcp port 443 -c 50 docker-compose logs caddy --tail=200 sudo ss -tunlp | grep -E '80|443' • Paste the tcpdump lines and the last 50 caddy log lines; I’ll identify whether requests are reaching Caddy and why ACME/TLS fails. 6) Confirm NLB and security groups • NLB listeners: TCP 80 → TG(port 80), TCP 443 → TG(port 443). Target groups: Instance type, targets = your EC2 instance(s). • NLB SG: inbound TCP 80/443 allowed from client CIDR (10.0.0.0/8). • EC2 SG: inbound TCP 80/443 allowed from NLB SG. • NACLs: allow 80/443 both directions (you said they are open). 7) After certs are obtained • Test WSS: wscat -c wss://livekitstaging.825382393.cloud.com/rtc • On the server confirm LiveKit sees connections and signaling succeeds. Notes/troubleshooting pointers • If Caddy logs ACME errors complaining about unreachable challenge, DNS likely not pointing to NLB or NLB not forwarding 80 properly. • If tcpdump shows packets arriving but Caddy logs "EOF", share the tcpdump + caddy logs and I’ll diagnose the EOF cause. • You already confirmed UDP media arrives — keep that unchanged. Do step 1–4 and paste the outputs requested in step 5 if it does not show "certificate obtained successfully". I’ll pinpoint the exact failure. Sources: Deploy to a VM | LiveKit docs | Self-hosting the Egress Service | LiveKit docs | Deploying LiveKit | LiveKit docs
f
unable to see your yaml, please display
t
Sorry about that — here is the corrected Caddy YAML (full, copy-pasteable) I recommended. It adds an HTTP :80 server for ACME challenges and TLS termination on :443 with SNI routing for LiveKit signaling and TURN.
Copy code
logging:
  logs:
    default:
      level: INFO

storage:
  module: file_system
  root: /data

apps:
  tls:
    certificates:
      automate:
        - <http://livekitstaging.825382393.cloud.com|livekitstaging.825382393.cloud.com>
        - <http://livekitstaging-turn.825382393.cloud.com|livekitstaging-turn.825382393.cloud.com>

  layer4:
    servers:
      https_frontend:
        listen: [":443"]
        routes:
          # Route for TURN/TLS (SNI-based routing to TURN server on 5349)
          - match:
              - tls:
                  sni:
                    - <http://livekitstaging-turn.825382393.cloud.com|livekitstaging-turn.825382393.cloud.com>
            handle:
              - handler: tls
              - handler: proxy
                upstreams:
                  - dial: ["10.148.214.166:5349"]

          # Route for LiveKit WSS (SNI-based routing to livekit on 7880)
          - match:
              - tls:
                  sni:
                    - <http://livekitstaging.825382393.cloud.com|livekitstaging.825382393.cloud.com>
            handle:
              - handler: tls
              - handler: proxy
                upstreams:
                  - dial: ["localhost:7880"]

  http:
    servers:
      acme_solver:
        listen: [":80"]
        routes:
          - match:
              - path: "/.well-known/acme-challenge/*"
            handle:
              - handler: file_server
                root: /data/acme-challenges
After replacing your current caddy.yaml with this file: 1. Create the ACME directory and set permissions: ◦ sudo mkdir -p /data/acme-challenges ◦ sudo chown -R 1000:1000 /data 1. Restart Caddy: ◦ docker-compose restart caddy 1. Verify: ◦ sudo ss -tunlp | grep -E '80|443' (Caddy should bind both) ◦ docker-compose logs -f caddy (watch for "certificate obtained successfully") If you want, paste the last 50 lines of the Caddy logs if certificates still don’t appear and I’ll help debug.
f
I am getting this error in caddy, caddy container keeps restarting {"level":"info","ts":1765883475.6952925,"msg":"using config from file","file":"/etc/caddy.yaml"} {"level":"info","ts":1765883475.696123,"msg":"adapted config to JSON","adapter":"yaml"} {"level":"info","ts":1765883475.6979237,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[:1]2019","//127.0.0.1:2019"]} {"level":"info","ts":1765883475.700147,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000646400"} {"level":"info","ts":1765883475.7003841,"logger":"http","msg":"servers shutting down with eternal grace period"} {"level":"info","ts":1765883475.7008305,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000646400"} {"level":"info","ts":1765883475.70104,"msg":"maxprocs: No GOMAXPROCS change to reset"} Error: loading initial config: loading new config: loading http app module: provision http: server acme_solver: setting up route matchers: route 0: loadingmatcher modules: module name 'path': decoding module config: http.matchers.path: json: cannot unmarshal string into Go value of type caddyhttp.MatchPath root@ip-10-148-214-14:/opt/livekit#
t
Good catch — the error shows Caddy failed to parse the 
path
matcher because I used a string instead of an array. Replace the HTTP section with this corrected snippet and restart Caddy. Use this exact HTTP block (paste into your caddy.yaml under apps):
Copy code
http:
    servers:
      acme_solver:
        listen: [":80"]
        routes:
          - match:
              - path:
                  - "/.well-known/acme-challenge/*"
            handle:
              - handler: file_server
                root: /data/acme-challenges
Steps to apply and verify 1. Update /etc/caddy.yaml with the corrected block. 2. Ensure ACME directory exists and permissions: sudo mkdir -p /data/acme-challenges sudo chown -R 1000:1000 /data 1. Restart Caddy: docker-compose restart caddy 1. Watch logs: docker-compose logs -f caddy Expected outcome • Caddy should start without the JSON unmarshal error. • You should see logs attempting ACME challenge and then "certificate obtained successfully" once HTTP 80 is reachable via the NLB/DNS. If it still restarts, paste the last 30 lines of the caddy logs and I’ll pinpoint the next issue.
f
your new code is not different from your previous code
t
Thanks — you’re right: the visible change was small (the path matcher must be an array). Let’s confirm and get Caddy stable. Do this exact replace in your /etc/caddy.yaml (complete apps.http block only):
Copy code
http:
    servers:
      acme_solver:
        listen: [":80"]
        routes:
          - match:
              - path:
                  - "/.well-known/acme-challenge/*"
            handle:
              - handler: file_server
                root: /data/acme-challenges
Then run these commands and paste the outputs if it still fails: 1) Ensure ACME dir & permissions sudo mkdir -p /data/acme-challenges sudo chown -R 1000:1000 /data 2) Restart caddy docker-compose restart caddy 3) Show caddy status/logs (paste last 40 lines) docker-compose logs --tail=40 caddy 4) Confirm listeners sudo ss -tunlp | grep -E '80|443' || true If the container still restarts, paste the logs from step 3 and the exact caddy.yaml file (full) so I can spot any remaining syntax issues.
f
this is caddy logs {"level":"info","ts":1765884044.6524723,"msg":"maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined"} {"level":"info","ts":1765884044.6526515,"msg":"GOMEMLIMIT is updated","package":"github.com/KimMachineGun/automemlimit/memlimit","GOMEMLIMIT":3691929600,"previous":9223372036854775807} {"level":"info","ts":1765884044.652689,"msg":"using config from file","file":"/etc/caddy.yaml"} {"level":"info","ts":1765884044.6534796,"msg":"adapted config to JSON","adapter":"yaml"} {"level":"info","ts":1765884044.6550875,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[:1]2019","//127.0.0.1:2019"]} {"level":"info","ts":1765884044.6559176,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0005d0780"} {"level":"warn","ts":1765884044.6560163,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"acme_solver","http_port":80} {"level":"warn","ts":1765884044.6567228,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"} {"level":"warn","ts":1765884044.656803,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"} {"level":"info","ts":1765884044.6568103,"logger":"http.log","msg":"server running","name":"acme_solver","protocols":["h1","h2","h3"]} {"level":"info","ts":1765884044.6608703,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"} {"level":"info","ts":1765884044.6608944,"msg":"serving initial configuration"} {"level":"info","ts":1765884044.6633804,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data"} {"level":"info","ts":1765884044.6688385,"logger":"tls","msg":"finished cleaning storage units"} this is livekit logs 2025-12-16T112044.749Z INFO livekit redis/redis.go:144 connecting to redis {"simple": true, "addr": "localhost:6379"} 2025-12-16T112044.806Z INFO livekit rtcconfig/ip.go:195 found external IP via STUN {"localAddr": "10.148.214.14:51229", "stunServer": "global.stun.twilio.com:3478", "externalIP": "3.125.234.252"} 2025-12-16T112044.806Z INFO livekit rtcconfig/ip.go:195 found external IP via STUN {"localAddr": "172.17.0.1:51229", "stunServer": "global.stun.twilio.com:3478", "externalIP": "3.125.234.252"} 2025-12-16T112049.753Z WARN livekit rtcconfig/ip.go:278 could not validate external IP {"ip": "3.125.234.252", "from": "10.148.214.14:51229", "error": "context canceled"} 2025-12-16T112049.753Z WARN livekit rtcconfig/ip.go:278 could not validate external IP {"ip": "3.125.234.252", "from": "172.17.0.1:51229","error": "context canceled"} 2025-12-16T112049.764Z INFO livekit rtcconfig/webrtc_config.go:284 failed to get external ip {"local": "10.148.214.14", "err": "context canceled"} 2025-12-16T112049.764Z INFO livekit rtcconfig/webrtc_config.go:284 failed to get external ip {"local": "172.17.0.1", "err": "context canceled"} 2025-12-16T112049.765Z INFO livekit rtcconfig/webrtc_config.go:99 no external IPs found, using node IP for NAT1To1Ips {"ip": "3.125.234.252"} 2025-12-16T112049.766Z INFO livekit service/turn.go:145 Starting TURN server {"turn.relay_range_start": 30000, "turn.relay_range_end": 40000, "turn.portTLS": 5349, "turn.externalTLS": true, "turn.portUDP": 3478} 2025-12-16T112049.770Z INFO livekit service/server.go:264 starting LiveKit server {"portHttp": 7880, "nodeID": "ND_aZffxpQmj2U7", "nodeIP": "3.125.234.252", "version": "1.9.8", "bindAddresses": [""], "rtc.portTCP": 7881, "rtc.portICERange": [50000, 60000]}
t
Thanks — logs are helpful. Two main issues remain: Caddy now runs and serves HTTP (so ACME can work), but LiveKit is detecting an external IP (3.125.234.252) via STUN and warning it can’t validate it. In a private NLB/VPC setup you must ensure LiveKit advertises the correct external address (the NLB address reachable by clients) so ICE candidates and WSS work. Do these steps in order: 1) Confirm what clients use to reach LiveKit • From a client inside your VPN, run: dig livekitstaging.825382393.cloud.com +short — note the IP or DNS (should be the NLB address reachable via VPN). • That address is what LiveKit must advertise (via STUN or explicit config). 2) Make LiveKit advertise the NLB address (pick one) Option A — Preferred: set explicit external IP/DNS in livekit.yaml • Edit /opt/livekit/livekit.yaml: rtc: port_range_start: 50000 port_range_end: 60000 use_external_ip: false external_ip: "<NLB_IP_OR_DNS>" # put the NLB’s internal IP (or DNS name) that clients reach turn: external_tls: true domain: livekitstaging-turn.825382393.cloud.com • Restart livekit container: docker-compose restart livekit Why: prevents STUN-detected public IP from being used; ensures advertised ICE candidates match the NLB endpoint. Option B — If you prefer STUN but must override NAT1:1 • Keep 
use_external_ip: true
but set 
nat_1to1: ["<NLB_IP>"]
or 
rtc.nat1to1_ips
(if your version supports it) to enforce the IP used for candidates. 3) Ensure Caddy / TLS flow matches NLB listeners • NLB should forward: ◦ TCP 80 → instance 80 (for ACME) ◦ TCP 443 → instance 443 (Caddy TLS terminate) ◦ UDP 3478 → instance 3478 ◦ UDP 50000–60000 → instance (port-preserve via UDP TG instance targets) or UDP 7882 if using UDP mux • Confirm target groups for 80 and 443 use Instance targets and port = 80 / 443 respectively. 4) Verify flow after config change • Restart LiveKit and Caddy: docker-compose restart livekit caddy • From a VPN client: ◦ curl -vk https://livekitstaging.825382393.cloud.com/ (should TLS handshake) ◦ wscat -c wss://livekitstaging.825382393.cloud.com/rtc (should connect) • On server: ◦ docker-compose logs livekit | tail -n 50 — verify LiveKit logs external IP = the NLB address you set ◦ sudo tcpdump -n -i any udp portrange 50000-60000 — confirm UDP media arrives 5) If agent still times out, capture and share • Last 50 lines of livekit logs around join attempt • A short tcpdump of signaling attempt (tcp port 443) and media attempt (udp portrange) • The client-side error message / stack trace of the timeout If you want, tell me the NLB IP/DNS you saw from dig and I’ll give the exact livekit.yaml snippet to paste (with proper key indentation). Which NLB IP/DNS did dig return?
f
This is output from VPN client (from my local machine) karthi.madhavan.ext@Mac-MXJGQQYXLM agent % dig livekitstaging.825382393.cloud.com +short 63.180.107.5 63.178.59.195 this is my livekit yaml, please change and provide port: 7880 bind_addresses: - "" rtc: tcp_port: 7881 port_range_start: 50000 port_range_end: 60000 use_external_ip: true enable_loopback_candidate: false redis: address: localhost:6379 username: "" password: "" db: 0 use_tls: false sentinel_master_name: "" sentinel_username: "" sentinel_password: "" sentinel_addresses: [] cluster_addresses: [] max_redirects: null turn: enabled: true domain: livekitstaging-turn.825382393.cloud.com tls_port: 5349 udp_port: 3478 external_tls: true keys: PIf5XLjbhFGNy: uKPUeKsjcJwVYfSgLdabAydh69JlM2HDbh1WU2JWC
t
Thanks — use this updated 
livekit.yaml
. It disables STUN autodiscovery and sets the external IP(s) that clients reach (your NLB IPs), so LiveKit will advertise correct ICE candidates. Replace your file with this (preserve indentation):
Copy code
port: 7880
bind_addresses:
  - ""
rtc:
  tcp_port: 7881
  port_range_start: 50000
  port_range_end: 60000
  use_external_ip: false
  external_ips:
    - "63.180.107.5"
    - "63.178.59.195"
  enable_loopback_candidate: false
redis:
  address: localhost:6379
  username: ""
  password: ""
  db: 0
  use_tls: false
  sentinel_master_name: ""
  sentinel_username: ""
  sentinel_password: ""
  sentinel_addresses: []
  cluster_addresses: []
  max_redirects: null
turn:
  enabled: true
  domain: <http://livekitstaging-turn.825382393.cloud.com|livekitstaging-turn.825382393.cloud.com>
  tls_port: 5349
  udp_port: 3478
  external_tls: true
keys:
  PIf5XLjbhFGNy: uKPUeKsjcJwVYfSgLdabAydh69JlM2HDbh1WU2JWC
Steps after replacing the file: 1. Restart services: ◦ docker-compose restart livekit caddy 1. Verify LiveKit advertises the configured external IPs: ◦ docker-compose logs livekit --tail=200 | grep -E "external IP|nat1to1" ◦ LiveKit startup logs should show the external IPs you set. 1. Test from a VPN client: ◦ wss connect: wscat -c wss://livekitstaging.825382393.cloud.com/rtc ◦ Start a call and confirm media flows; on the server run: sudo tcpdump -n -i any udp portrange 50000-60000 1. If join still times out, collect and share: ◦ Last 100 lines of livekit logs around the join attempt ◦ tcpdump capture showing signaling (tcp port 443) and media (udp portrange) during the attempt If you prefer, I can also provide a variant that uses 
nat_1to1_ips
instead of 
external_ips
depending on your LiveKit version — tell me which LiveKit version you run if you want that.
f
livekit server restarting with below error root@ip-10-148-214-14:/opt/livekit# docker logs -f 9c could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field external_ips not found in type config.RTCConfig
t
Thanks — the error means your LiveKit version doesn’t recognize 
external_ips
. Use the supported field 
nat_1to1_ips
(or set 
use_external_ip: false
and 
external_ip
if supported). Let’s fix the file to use 
nat_1to1_ips
, which works on current LiveKit releases. Replace your rtc section with this exact block:
Copy code
rtc:
  tcp_port: 7881
  port_range_start: 50000
  port_range_end: 60000
  use_external_ip: false
  nat_1to1_ips:
    - "63.180.107.5"
    - "63.178.59.195"
  enable_loopback_candidate: false
Full file for copy/paste (replace your current livekit.yaml):
Copy code
port: 7880
bind_addresses:
  - ""
rtc:
  tcp_port: 7881
  port_range_start: 50000
  port_range_end: 60000
  use_external_ip: false
  nat_1to1_ips:
    - "63.180.107.5"
    - "63.178.59.195"
  enable_loopback_candidate: false
redis:
  address: localhost:6379
  username: ""
  password: ""
  db: 0
  use_tls: false
  sentinel_master_name: ""
  sentinel_username: ""
  sentinel_password: ""
  sentinel_addresses: []
  cluster_addresses: []
  max_redirects: null
turn:
  enabled: true
  domain: <http://livekitstaging-turn.825382393.cloud.com|livekitstaging-turn.825382393.cloud.com>
  tls_port: 5349
  udp_port: 3478
  external_tls: true
keys:
  PIf5XLjbhFGNy: uKPUeKsjcJwVYfSgLdabAydh69JlM2HDbh1WU2JWC
After updating: 1. Save file. 2. Restart LiveKit: docker-compose restart livekit 3. Check logs: docker logs -f <livekit_container_id> and confirm startup shows NAT 1:1 IPs or external IPs matching the two addresses. 4. Test from VPN client: attempt join; on EC2 run tcpdump for UDP portrange and tcp port 443 while reproducing. If you still see a parsing error, paste the LiveKit container startup log lines (first ~30) and the exact LiveKit image version (docker image tag shown by docker-compose). I’ll adjust the config to the exact version fields.
f
again container keep restarting with same error could not parse config: yaml: unmarshal errors: line 9: field nat_1to1_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field nat_1to1_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field nat_1to1_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field nat_1to1_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field nat_1to1_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field nat_1to1_ips not found in type config.RTCConfig could not parse config: yaml: unmarshal errors: line 9: field nat_1to1_ips not found in type config.RTCConfig
t
Thanks — before changing the config again, I need two quick facts so I don’t suggest unsupported fields that crash your container. 1) What LiveKit Docker image/tag are you running? • Run: docker inspect --format='{{.Config.Image}}' <livekit_container_id> • Or paste the image name shown in your docker-compose (example: livekit/livekit-server:1.9.8) 2) Paste the first 30 lines of the livekit container logs (start of current restart cycle): • docker logs --tail=30 <livekit_container_id> Once I have those I will: • Confirm which rtc fields your version supports, and • Provide a safe config change (or minimal revert to 
use_external_ip: true
) that will not cause parsing errors, plus the exact commands to apply and verify. Sources: Deploying LiveKit | LiveKit docs | Ports and firewall | LiveKit docs
f
i am using livekit/livekit-server:latest
t
Thanks — since you’re using the 
latest
image and several attempted fields are unsupported by that runtime, revert to a minimal, supported RTC config so the server starts, then we’ll apply the proper external-IP approach that works for your version. 1) Replace your livekit.yaml with this safe, supported config (copy-paste exactly):
Copy code
port: 7880
bind_addresses:
  - ""
rtc:
  tcp_port: 7881
  port_range_start: 50000
  port_range_end: 60000
  use_external_ip: true
  enable_loopback_candidate: false
redis:
  address: localhost:6379
  username: ""
  password: ""
  db: 0
  use_tls: false
turn:
  enabled: true
  domain: <http://livekitstaging-turn.825382393.cloud.com|livekitstaging-turn.825382393.cloud.com>
  tls_port: 5349
  udp_port: 3478
  external_tls: true
keys:
  PIf5XLjbhFGNy: uKPUeKsjcJwVYfSgLdabAydh69JlM2HDbh1WU2JWC
Why: 
use_external_ip: true
is supported and avoids YAML parse errors from unsupported fields like 
external_ips
or 
nat_1to1_ips
. This lets LiveKit use STUN to discover an external IP while avoiding crashes. 2) Restart and verify startup • Run: docker-compose restart livekit docker logs -f <livekit_container_id> • Expected: LiveKit should start without "could not parse config" errors and log STUN/external IP discovery messages (no crash loop). 3) If LiveKit starts but clients still can’t join, we have two follow-ups depending on what you want: A — Prefer STUN autodiscovery (current): keep 
use_external_ip: true
• Verify the external IP LiveKit reports matches an IP reachable by your clients (the two IPs you saw from dig). • If STUN returns a public IP not reachable via your VPN, we’ll instead configure NAT mapping—see B. B — Prefer fixed advertised addresses (when STUN autodiscovery is unsuitable) • We need the exact LiveKit server version (image digest/tag) to pick the correct supported field. You can get it with: docker inspect --format='{{.Config.Image}}' <livekit_container_id> • If your image supports setting a static advertised IP, I’ll provide the exact config syntax for that version. 4) Quick tests to run after the server is up • From an inside-VPN client: ◦ dig livekitstaging.825382393.cloud.com +short ◦ curl -vk https://livekitstaging.825382393.cloud.com/ ◦ wscat -c wss://livekitstaging.825382393.cloud.com/rtc • On server: ◦ docker-compose logs livekit --tail=200 ◦ sudo tcpdump -n -i any udp portrange 50000-60000 (during a call attempt) ◦ sudo tcpdump -n -i any tcp port 443 (during a join attempt) 5) If you still hit join timeouts, paste: • The livekit startup logs (first ~50 lines after restart) • A short snippet of livekit logs during the join attempt • Results of the tcpdump commands above I’ll then identify whether the failure is: signaling (WSS/TLS), ICE candidate mismatch (external IP), TURN allocation issue, or UDP reachability. Which do you want to do next — restart with the config above and share the livekit startup logs, or have me provide the static-IP config after you confirm the exact image tag? Sources: Deploying LiveKit | LiveKit docs | Deploy to a VM | LiveKit docs
f
Now livekit server started but still domain url is not accessible and caddy logs also showing previous error only it is not obtaining any certificates
t
Thanks — let’s fix Caddy certificate provisioning and domain reachability step‑by‑step. You already have LiveKit running and UDP media arriving, so we only need to get HTTP/ACME and TLS correct. Follow these steps in order and paste the requested outputs only if a step fails. 1) Confirm DNS → NLB • From your VPN client run: dig livekitstaging.825382393.cloud.com +short • Expected: the NLB IP(s) you saw (63.180.107.5 and 63.178.59.195). If these match, continue. If not, update DNS to point to the NLB. 2) Confirm NLB listeners & target groups • Ensure listeners: ◦ TCP 80 → TG (instance targets, port 80) ◦ TCP 443 → TG (instance targets, port 443) • Ensure target groups are Instance type and instances show Healthy. • If any listener or TG is missing or shows Unhealthy, tell me which one. 3) Confirm security groups & NACLs • NLB SG: allow inbound TCP 80 and TCP 443 from client/VPN CIDR (10.0.0.0/8). • EC2 SG: allow inbound TCP 80 and TCP 443 from NLB SG. • Subnet NACLs: allow 80/443 both directions (you said they are open). • You said EC2 SG allows all traffic from NLB SG — that’s fine for testing. 4) Verify Caddy is listening on :80 and :443 • On EC2 run: sudo ss -tunlp | grep -E '80|443' • Expected output includes caddy bound to both :80 and :443. • You already showed caddy listening on :443 — confirm :80 appears. If :80 is not listed, paste the output. 5) Test HTTP reachability from VPN client • From your VPN client run: curl -vk http://livekitstaging.825382393.cloud.com/ • Possible results: ◦ If you get a successful HTTP response (200/301/404), proceed to step 6. ◦ If you get timeout: NLB/SG/DNS routing issue — paste the curl output. ◦ If you get 502 Bad Gateway: proceed to step 5a. 5a) If curl returns 502 Bad Gateway • On EC2 collect: ◦ sudo tcpdump -n -i any tcp port 80 -c 50 ◦ docker-compose logs --tail=100 caddy • Paste the last 30 lines of caddy logs and 10 lines of tcpdump output (I’ll analyze why Caddy returns 502). 6) Confirm Caddy can perform ACME (outbound) • On EC2 run: curl -v https://acme-v02.api.letsencrypt.org/directory • If this fails, the instance cannot reach Let’s Encrypt—ensure outbound internet (NAT gateway) is available. If it succeeds, continue. 7) Watch Caddy logs while you hit HTTP • On EC2: docker-compose logs -f caddy • From VPN client run the curl in step 5 again and watch logs. Look for ACME challenge logs or errors. If you do not see ACME challenge requests come in, the HTTP request is not reaching Caddy — paste the curl and caddy logs. 8) If ACME challenge reaches Caddy but certificate not issued • Paste the caddy log lines around the ACME attempt. I’ll identify the error (e.g., rate limit, challenge path issue, certificate authority error). 9) Final TLS verification • Once caddy logs show "certificate obtained successfully", verify: curl -vk https://livekitstaging.825382393.cloud.com/ wscat -c wss://livekitstaging.825382393.cloud.com/rtc If you prefer, start at step 4 and paste these three outputs now: • sudo ss -tunlp | grep -E '80|443' • curl -vk http://livekitstaging.825382393.cloud.com/ • docker-compose logs --tail=50 caddy I’ll analyze them and give the exact change to make next.
f
root@ip-10-148-214-14:/opt/livekit#sudo tcpdump -n -i any tcp port 80 -c 50y tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes packets not receiving and when checking curl command below is the output (502 gateway) karthi.madhavan.ext@Mac-MXJGQQYXLM agent % curl -vk https://livekitstaging.825382393065.cloud.bayer.com * Host livekitstaging.825382393065.cloud.bayer.com:443 was resolved. * IPv6: (none) * IPv4: 63.178.59.195, 63.180.107.5 * Trying 63.178.59.195:443... * Connected to livekitstaging.825382393065.cloud.bayer.com (63.178.59.195) port 443 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF * ALPN: server did not agree on a protocol. Uses default. * Server certificate: * subject: CN=livekitstaging.825382393065.cloud.bayer.com * start date: Dec 16 113713 2025 GMT * expire date: Dec 16 113713 2026 GMT * issuer: CN=BAYER InternetProxy; O=BAYER; OU=BBS; L=Leverkusen; ST=NRW; C=DE * SSL certificate verify ok. * using HTTP/1.x
GET / HTTP/1.1
Host: livekitstaging.825382393065.cloud.bayer.com
User-Agent: curl/8.7.1
Accept: /
* Request completely sent off < HTTP/1.1 502 cannotconnect < Connection: Keep-Alive < Content-Type: text/html < Cache-Control: no-cache < Content-Length: 5433 < X-Frame-Options: deny < <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <!-- FileName: index.html Language: [en] --> <html> <style type=text/css> <!-- a.tooltip, a.tooltip:link, a.tooltip:visited, a.tooltip:active { position: absolut; text-decoration: none; font-style: italic; color: #0000FF; } a.tooltip1, a.tooltip1:link, a.tooltip1:visited, a.tooltip1:active { position: absolut; text-decoration: none; font-style: italic; color: #0000FF; } a.tooltip:hover { color: #0000FF; background: transparent; } a.tooltip span { display: none; text-decoration: none; } a.tooltip:hover span { display: block; position: absolute; top: 180px; left: 150px; width: 900px; z-index: 100; color: #000000; border:1px solid #000000; background: #FFFFCC; font: 12px Verdana, sans-serif; text-align: left; } a.tooltip1 span b { display: block; margin: 0; padding: 0; font-size: 16px; font-weigth: bold; color: yellow; background-color: gray; border: 0px; border-bottom: 1px solid black; } a.tooltip1 span { display: none; text-decoration: none; } a.tooltip1:hover span { display: block; position: absolute; top: 0px; left: 0px; width: 900px; z-index: 100; color: #000000; border:1px solid #000000; background: #FFFFCC; font: 12px Verdana, sans-serif; text-align: left; } a.tooltip1 span b { display: block; margin: 0; padding: 0; font-size: 16px; font-weigth: bold; color: yellow; background-color: gray; border: 0px; border-bottom: 1px solid black; }
t
Thanks — helpful traces. The curl TLS handshake shows a corporate TLS proxy (issuer CN=BAYER InternetProxy) returning a 502 HTML page — that means your client’s HTTPS request is being intercepted by an enterprise proxy before it reaches the NLB/instance. That explains why tcpdump on EC2 saw no port 80 packets. What this implies • The client traffic is not reaching your NLB because the corporate proxy terminates or redirects requests locally. • ACME and Caddy’s HTTP challenge will also be blocked from that client path, so certificate issuance and WSS testing from that client will fail while the proxy is in place. Actionable next steps (do these in order) 1) Verify client path reaches NLB (bypass proxy) • From a machine on the same VPN that your agents use, confirm no corporate HTTPS proxy is intercepting: ◦ Temporarily disable the corporate proxy in your browser or curl (or use curl --noproxy '*') and retry: curl -vk --noproxy '*' https://livekitstaging.825382393065.cloud.bayer.com/ • If that returns the Caddy/TLS response (or different error), the proxy was the blocker. 2) Test from inside the VPC (definitive) • From a bastion host or another EC2 instance in the same VPC/subnet, run: curl -vk http://<your-domain>/ curl -vk https://<your-domain>/ • Also run tcpdump on your LiveKit host while doing these tests to confirm packets arrive. 3) If proxy must remain in path, arrange one of: • Allowlist the domain through the corporate proxy so traffic passes to the NLB (speak with your network/security team). • Use an internal DNS name that resolves to the NLB internal IP and make clients use that over VPN (bypass external web proxy). • Use corporate proxy settings to allow CONNECT passthrough to your domain (requires proxy config change). 4) Validate NLB & TG health one more time • In AWS console confirm: ◦ NLB listeners: TCP 80 → TG(port 80) and TCP 443 → TG(port 443). ◦ Target groups show your instance Healthy for those ports. • If any TG is Unhealthy, paste the TG health check and target status; I’ll advise fixes. 5) After bypassing proxy or allowlisting, re-run: • On EC2: sudo tcpdump -n -i any tcp port 80 or 443 • From client (no-proxy or internal VPC): curl -vk http://... and watch tcpdump and Caddy logs • Expect ACME challenge hits and “certificate obtained successfully” in Caddy logs. If you want, I can: • Provide the exact curl commands you should run to bypass proxy and test. • Draft a short note you can send to your network/security team asking them to allow CONNECT to your domain or add the domain to the proxy allowlist. Which do you want to do next — run the no-proxy curl test now, or shall I draft the allowlist request for your security team?
f
I am receiving packets also in ec2 }root@ip-10-148-214-14:/opt/livekit#sudo tcpdump -n -i any tcp port 80 -c 50y tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 114715.170398 enX0 Out IP 10.148.214.14.35326 > 169.254.169.254.80: Flags [S], seq 3694772559, win 62727, options [mss 8961,sackOK,TS val 2258116937 ecr0,nop,wscale 7], length 0 114715.170651 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35326: Flags [S.], seq 1685795650, ack 3694772560, win 62643, options [mss 8961,sackOK,TS val 7254776 ecr 2258116937,nop,wscale 7], length 0 114715.170690 enX0 Out IP 10.148.214.14.35326 > 169.254.169.254.80: Flags [.], ack 1, win 491, options [nop,nop,TS val 2258116937 ecr 7254776], length 0 114715.170855 enX0 Out IP 10.148.214.14.35326 > 169.254.169.254.80: Flags [P.], seq 1:247, ack 1, win 491, options [nop,nop,TS val 2258116938 ecr 7254776], length 246: HTTP: GET /latest/meta-data/iam/security-credentials/ HTTP/1.1 114715.171051 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35326: Flags [.], ack 247, win 488, options [nop,nop,TS val 7254777 ecr 2258116938], length0 114715.172129 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35326: Flags [P.], seq 1:274, ack 247, win 488, options [nop,nop,TS val 7254778 ecr 2258116938], length 273: HTTP: HTTP/1.0 200 OK 114715.172147 enX0 Out IP 10.148.214.14.35326 > 169.254.169.254.80: Flags [.], ack 274, win 489, options [nop,nop,TS val 2258116939 ecr 7254778], length0 114715.172157 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35326: Flags [F.], seq 274, ack 247, win 488, options [nop,nop,TS val 7254778 ecr 2258116938], length 0 114715.172261 enX0 Out IP 10.148.214.14.35326 > 169.254.169.254.80: Flags [F.], seq 247, ack 275, win 489, options [nop,nop,TS val 2258116939 ecr 7254778], length 0 114715.172370 enX0 Out IP 10.148.214.14.35332 > 169.254.169.254.80: Flags [S], seq 1230699716, win 62727, options [mss 8961,sackOK,TS val 2258116939 ecr0,nop,wscale 7], length 0 114715.172432 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35326: Flags [.], ack 248, win 488, options [nop,nop,TS val 7254778 ecr 2258116939], length0 114715.172527 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35332: Flags [S.], seq 235581115, ack 1230699717, win 62643, options [mss 8961,sackOK,TS val 7254778 ecr 2258116939,nop,wscale 7], length 0 114715.172546 enX0 Out IP 10.148.214.14.35332 > 169.254.169.254.80: Flags [.], ack 1, win 491, options [nop,nop,TS val 2258116939 ecr 7254778], length 0 114715.172633 enX0 Out IP 10.148.214.14.35332 > 169.254.169.254.80: Flags [P.], seq 1:272, ack 1, win 491, options [nop,nop,TS val 2258116939 ecr 7254778], length 271: HTTP: GET /latest/meta-data/iam/security-credentials/mon-default-instance-role HTTP/1.1 114715.172803 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35332: Flags [.], ack 272, win 488, options [nop,nop,TS val 7254779 ecr 2258116939], length0 114715.173255 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35332: Flags [P.], seq 1:1853, ack 272, win 488, options [nop,nop,TS val 7254779 ecr 2258116939], length 1852: HTTP: HTTP/1.0 200 OK 114715.173272 enX0 Out IP 10.148.214.14.35332 > 169.254.169.254.80: Flags [.], ack 1853, win 477, options [nop,nop,TS val 2258116940 ecr 7254779], length 0 114715.173285 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35332: Flags [F.], seq 1853, ack 272, win 488, options [nop,nop,TS val 7254779 ecr 2258116939], length 0 114715.173370 enX0 Out IP 10.148.214.14.35332 > 169.254.169.254.80: Flags [F.], seq 272, ack 1854, win 477, options [nop,nop,TS val 2258116940 ecr 7254779], length 0 114715.173523 enX0 In IP 169.254.169.254.80 > 10.148.214.14.35332: Flags [.], ack 273, win 488, options [nop,nop,TS val 7254779 ecr 2258116940], length0 114811.975112 enX0 Out IP 10.148.214.14.43416 > 169.254.169.254.80: Flags [S], seq 2453425866, win 62727, options [mss 8961,sackOK,TS val 2258173742 ecr0,nop,wscale 7], length 0 114811.975339 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43416: Flags [S.], seq 534422352, ack 2453425867, win 62643, options [mss 8961,sackOK,TS val 7311581 ecr 2258173742,nop,wscale 7], length 0 114811.975376 enX0 Out IP 10.148.214.14.43416 > 169.254.169.254.80: Flags [.], ack 1, win 491, options [nop,nop,TS val 2258173742 ecr 7311581], length 0 114811.975511 enX0 Out IP 10.148.214.14.43416 > 169.254.169.254.80: Flags [P.], seq 1:201, ack 1, win 491, options [nop,nop,TS val 2258173742 ecr 7311581], length 200: HTTP: PUT /latest/api/token HTTP/1.1 114811.975771 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43416: Flags [.], ack 201, win 488, options [nop,nop,TS val 7311582 ecr 2258173742], length0 114811.977089 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43416: Flags [P.], seq 1:238, ack 201, win 488, options [nop,nop,TS val 7311583 ecr 2258173742], length 237: HTTP: HTTP/1.0 200 OK 114811.977103 enX0 Out IP 10.148.214.14.43416 > 169.254.169.254.80: Flags [.], ack 238, win 490, options [nop,nop,TS val 2258173744 ecr 7311583], length0 114811.977170 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43416: Flags [F.], seq 238, ack 201, win 488, options [nop,nop,TS val 7311583 ecr 2258173742], length 0 114811.977225 enX0 Out IP 10.148.214.14.43416 > 169.254.169.254.80: Flags [F.], seq 201, ack 239, win 490, options [nop,nop,TS val 2258173744 ecr 7311583], length 0 114811.977351 enX0 Out IP 10.148.214.14.43426 > 169.254.169.254.80: Flags [S], seq 1296155511, win 62727, options [mss 8961,sackOK,TS val 2258173744 ecr0,nop,wscale 7], length 0 114811.977449 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43416: Flags [.], ack 202, win 488, options [nop,nop,TS val 7311583 ecr 2258173744], length0 114811.977559 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43426: Flags [S.], seq 1095372522, ack 1296155512, win 62643, options [mss 8961,sackOK,TS val 7311583 ecr 2258173744,nop,wscale 7], length 0 114811.977577 enX0 Out IP 10.148.214.14.43426 > 169.254.169.254.80: Flags [.], ack 1, win 491, options [nop,nop,TS val 2258173744 ecr 7311583], length 0 114811.977670 enX0 Out IP 10.148.214.14.43426 > 169.254.169.254.80: Flags [P.], seq 1:233, ack 1, win 491, options [nop,nop,TS val 2258173744 ecr 7311583], length 232: HTTP: GET /latest/meta-data/instance-id HTTP/1.1 114811.977898 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43426: Flags [.], ack 233, win 488, options [nop,nop,TS val 7311584 ecr 2258173744], length0 114811.978868 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43426: Flags [P.], seq 1:269, ack 233, win 488, options [nop,nop,TS val 7311585 ecr 2258173744], length 268: HTTP: HTTP/1.0 200 OK 114811.978887 enX0 Out IP 10.148.214.14.43426 > 169.254.169.254.80: Flags [.], ack 269, win 489, options [nop,nop,TS val 2258173746 ecr 7311585], length0 114811.978947 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43426: Flags [F.], seq 269, ack 233, win 488, options [nop,nop,TS val 7311585 ecr 2258173744], length 0 114811.978970 enX0 Out IP 10.148.214.14.43426 > 169.254.169.254.80: Flags [F.], seq 233, ack 270, win 489, options [nop,nop,TS val 2258173746 ecr 7311585], length 0 114811.979095 enX0 Out IP 10.148.214.14.43428 > 169.254.169.254.80: Flags [S], seq 3375737415, win 62727, options [mss 8961,sackOK,TS val 2258173746 ecr0,nop,wscale 7], length 0 114811.979160 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43426: Flags [.], ack 234, win 488, options [nop,nop,TS val 7311585 ecr 2258173746], length0 114811.979271 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43428: Flags [S.], seq 2085236487, ack 3375737416, win 62643, options [mss 8961,sackOK,TS val 7311585 ecr 2258173746,nop,wscale 7], length 0 114811.979287 enX0 Out IP 10.148.214.14.43428 > 169.254.169.254.80: Flags [.], ack 1, win 491, options [nop,nop,TS val 2258173746 ecr 7311585], length 0 114811.979382 enX0 Out IP 10.148.214.14.43428 > 169.254.169.254.80: Flags [P.], seq 1:246, ack 1, win 491, options [nop,nop,TS val 2258173746 ecr 7311585], length 245: HTTP: GET /latest/dynamic/instance-identity/document HTTP/1.1 114811.979561 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43428: Flags [.], ack 246, win 488, options [nop,nop,TS val 7311585 ecr 2258173746], length0 114811.980193 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43428: Flags [P.], seq 1:735, ack 246, win 488, options [nop,nop,TS val 7311586 ecr 2258173746], length 734: HTTP: HTTP/1.0 200 OK 114811.980207 enX0 Out IP 10.148.214.14.43428 > 169.254.169.254.80: Flags [.], ack 735, win 486, options [nop,nop,TS val 2258173747 ecr 7311586], length0 114811.980263 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43428: Flags [F.], seq 735, ack 246, win 488, options [nop,nop,TS val 7311586 ecr 2258173746], length 0 114811.980319 enX0 Out IP 10.148.214.14.43428 > 169.254.169.254.80: Flags [F.], seq 246, ack 736, win 486, options [nop,nop,TS val 2258173747 ecr 7311586], length 0 114811.980477 enX0 In IP 169.254.169.254.80 > 10.148.214.14.43428: Flags [.], ack 247, win 488, options [nop,nop,TS val 7311586 ecr 2258173747], length0 50 packets captured 82 packets received by filter 0 packets dropped by kernel please help where the issue is?
t
Thanks — the tcpdump you pasted shows only traffic to the EC2 metadata service (169.254.169.254), not HTTP requests from your client. That means HTTP from your VPN client is not reaching the instance on port 80. Let’s narrow the gap with a few quick checks and actions. 1) Confirm NLB listener → target group mapping • In the AWS Console: Load Balancers → select your NLB → Listeners. • Verify there is a listener: TCP 80 → Target group A (Instance targets, port 80). • Share the exact target group name for the TCP 80 listener if you want me to validate. 2) Confirm target group health and port • EC2 → Target Groups → open the TG used by the 80 listener. • Check: ◦ Target type = Instance ◦ Port = 80 ◦ Health check protocol = TCP, port = 80 ◦ Targets: your instance listed and Health = Healthy • If targets are Unhealthy, that explains 502 — paste the TG health status and recent health check failures. 3) Confirm security groups and NACLs • NLB SG inbound: allow TCP 80 from your VPN CIDR (10.0.0.0/8) — you said this is set. • EC2 SG inbound: allow TCP 80 from NLB SG (you said EC2 SG allows NLB SG all traffic) — that’s OK for testing. • Subnet NACLs: you said they allow 0.0.0.0/0 — OK. 4) Test reachability from inside the VPC (definitive) • From another EC2 in same VPC (bastion) or from the instance itself use the NLB IP: ◦ curl -vk http://63.178.59.195:80/ ◦ curl -vk http://63.180.107.5:80/ • While running the curl, on the LiveKit host run: sudo tcpdump -n -i any tcp port 80 -c 20 • If these curls hit the instance and appear in tcpdump, the path from NLB → instance is OK and the issue is on the client side (proxy/DNS). If they do NOT appear, the NLB → TG mapping or TG health is the problem. 5) If curls to NLB IPs succeed but your VPN client still gets 502: • Your corporate proxy is likely intercepting/terminating requests from the client. Test from client with: curl -vk --noproxy '*' https://livekitstaging.825382393065.cloud.bayer.com/ • If that works, ask your network team to allow CONNECT/forwarding for that domain or add it to the proxy allowlist. 6) If curls to NLB IPs do NOT reach the instance • Share these screenshots/text: ◦ NLB Listeners table (showing listener protocol/port → TG) ◦ Target Group details for the port-80 TG (health check config and target health) ◦ NLB Attributes if you changed cross-zone or preserve client ip I’ll read them and point to the exact misconfiguration. Next step recommendation • Run the two curl tests to the NLB IPs from a host inside the VPC (step 4) and paste results plus the TG health status. I’ll analyze and give the exact fix.
f
1,2 is correct same configuration i have, and all listerner health is good 4. i tested that curl command from ec2 not getting any output its hanging
t
Thanks — helpful context. I’ll guide you through two short, targeted tests so I can see exactly where the NLB→instance path is failing. Please run them and paste the outputs. Test A — from a second EC2 in the same VPC (preferred) 1. On the LiveKit host (start this first): sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 40 (leave this running) 1. From a different EC2 in same VPC/subnet (not the LiveKit instance) run: curl -v http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' 1. Stop tcpdump after curl finishes and paste the first ~40 tcpdump lines. Test B — from the LiveKit instance itself (to check hairpin) 1. On LiveKit host (start tcpdump): sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 40 1. In another SSH session on the same LiveKit host run: curl -v http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' 1. Paste the curl output and tcpdump lines. Also paste these from the LiveKit host: • sudo ss -tunlp | grep -E '80|443' • sudo ip route show • sudo sysctl net.ipv4.conf.all.rp_filter I’ll analyze the captures and outputs and tell you exactly which NLB/TG/SG/routing change to make.
f
This command is hanging in livekit ec2 not providing any output sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 40 tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes root@ip-10-148-214-14:/var/snap/amazon-ssm-agent/12322# sudo ss -tunlp | grep -E '80|443' tcp LISTEN 0 4096 *:443 : users:(("caddy",pid=17191,fd=7)) tcp LISTEN 0 4096 *:80 : users:(("caddy",pid=17191,fd=8)) root@ip-10-148-214-14:/var/snap/amazon-ssm-agent/12322# sudo ip route show default via 10.148.214.1 dev enX0 proto dhcp src 10.148.214.14 metric 100 10.148.214.0/26 dev enX0 proto kernel scope link src 10.148.214.14 metric 100 10.148.214.1 dev enX0 proto dhcp scope link src 10.148.214.14 metric 100 10.148.214.2 dev enX0 proto dhcp scope link src 10.148.214.14 metric 100 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown root@ip-10-148-214-14:/var/snap/amazon-ssm-agent/12322# sudo sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.all.rp_filter = 2
t
Thanks — useful info. You’ve shown Caddy listening on 80/443 and the instance routing looks normal, but requests to the NLB IP are not reaching the instance (tcpdump shows only metadata traffic). To find exactly where packets are dropped, please run these two short tests and paste their outputs. Test 1 — from the LiveKit host (hairpin test) 1. On the LiveKit host start tcpdump in one SSH session: sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 20 1. In another SSH session on the same host run: curl -v --noproxy '*' http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' 1. Paste both the tcpdump lines and the curl output. Test 2 — from a different EC2 in the same VPC (preferred) 1. On LiveKit host start tcpdump: sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 20 1. From a different EC2 in the same VPC/subnet, run: curl -v http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' 1. Paste both tcpdump lines and the curl output. Also paste these AWS items (copy/paste text is fine) • Is the NLB Scheme "internal" or "internet-facing"? • NLB DNS name (the full DNS) • The Target Group ARN used by the TCP 80 listener (from the NLB listeners view) • Output of: aws elbv2 describe-target-health --target-group-arn <TG-arn-for-80> Why these tests • If hairpin curl from the LiveKit host appears in tcpdump, NLB → instance path is OK and the issue is client-side (proxy/DNS/VPN). • If hairpin curl does NOT appear, NLB → instance routing or NLB attributes are blocking hairpin; the separate-EC2 test will confirm whether the problem is hairpin-specific or general NLB→instance. Do the two tests and paste the outputs; I’ll tell you exactly what to change next.
f
Please find my findings i executed this command in livekit host sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 20 when i execute this command from curl -v --noproxy '*' http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' same machine packets receiving below is the output, even i tested with sudo tcpdump -n -i any host livekitstaging.825382393065.cloud.bayer.com and tcp port 80 -c 20 root@ip-10-148-214-14:/var/snap/amazon-ssm-agent/12322# sudo tcpdump -n -i any host livekitstaging.825382393065.cloud.bayer.com and tcp port 80 -c 20 tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 120747.654815 enX0 Out IP 10.148.214.14.58360 > 63.178.59.195.80: Flags [S], seq 1185389411, win 62727, options [mss 8961,sackOK,TS val 3702469591 ecr 0,nop,wscale 7], length 0 120757.616966 enX0 Out IP 10.148.214.14.54366 > 63.178.59.195.80: Flags [S], seq 4198389739, win 62727, options [mss 8961,sackOK,TS val 3702479553 ecr 0,nop,wscale 7], length 0 120758.662828 enX0 Out IP 10.148.214.14.54366 > 63.178.59.195.80: Flags [S], seq 4198389739, win 62727, options [mss 8961,sackOK,TS val 3702480599 ecr 0,nop,wscale 7], length 0 120759.686812 enX0 Out IP 10.148.214.14.54366 > 63.178.59.195.80: Flags [S], seq 4198389739, win 62727, options [mss 8961,sackOK,TS val 3702481623 ecr 0,nop,wscale 7], length 0 120800.710762 enX0 Out IP 10.148.214.14.54366 > 63.178.59.195.80: Flags [S], seq 4198389739, win 62727, options [mss 8961,sackOK,TS val 3702482647 ecr 0,nop,wscale 7], length 0 120801.734806 enX0 Out IP 10.148.214.14.54366 > 63.178.59.195.80: Flags [S], seq 4198389739, win 62727, options [mss 8961,sackOK,TS val 3702483671 ecr 0,nop,wscale 7], length 0 from VPN client i am getting below output karthi.madhavan.ext@Mac-MXJGQQYXLM agent % curl -v --noproxy '*' http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 -- -- -- -- -- -- 0* Trying 63.178.59.195:80... * Connected to 63.178.59.195 (63.178.59.195) port 80
GET / HTTP/1.1
Host: 63.178.59.195
User-Agent: curl/8.7.1
Accept: /
* Request completely sent off 0 0 0 0 0 0 0 0 -- -- 00048 -- -- 0< HTTP/1.1 502 cannotconnect < content-type: text/html < cache-control: no-cache < content-length: 5262 < x-frame-options: deny < proxy-connection: Keep-Alive < strict-transport-security: max-age=31536000 < { [5262 bytes data] 100 5262 100 5262 0 0 108 0 00048 00048 -- -- 1214 * Connection #0 to host 63.178.59.195 left intact <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <!-- FileName: index.html Language: [en]
t
Thanks — your tests show two clear behaviors: • From the LiveKit EC2 itself your curl to the NLB IP produces outbound SYNs but no SYN‑ACKs captured by tcpdump (only outbound SYNs shown). That means hairpinning (instance → NLB → instance) is not completing. • From the VPN client you reach the NLB IP and see a 502 HTML proxy page — that proxy content indicates a corporate HTTP proxy is in the client path for some requests (but your no-proxy curl did reach the NLB IP and returned 502, so the NLB is reachable from the VPN client). Next focused actions — run these in order and report results for the single step that fails. They will let us pinpoint whether the NLB or AWS routing is blocking return traffic or hairpin behaviour. 1) Confirm NLB scheme and cross-zone setting • In AWS Console: EC2 → Load Balancers → select your NLB ◦ Note: Scheme = internal or internet-facing? ◦ Attributes → Cross-zone load balancing = Enabled? Reply with the values. 2) Test connectivity from a separate EC2 in the same VPC (not the LiveKit instance) • Launch/SSH another small EC2 in same subnet/AZ (or use an existing bastion). • From that host run: curl -v http://63.178.59.195:80/ (use the same NLB IP you tested) • Meanwhile on LiveKit instance run: sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 40 If tcpdump on the LiveKit host shows inbound SYN/ACK and the separate EC2 curl returns a response, NLB→instance path is good for external clients and the hairpin issue is limited to instance hairpinning. Paste tcpdump and curl outputs. 3) If separate EC2 curl DOES NOT reach instance • Paste the NLB Listener → Target Group mapping (the exact TG name/ARN) and the output: aws elbv2 describe-target-health --target-group-arn <TG-ARN> • Also paste the NLB DNS name and the TG health details. If target health is Healthy on AWS but packets still don't arrive, we’ll check subnet routing or ENI attachment issues. 4) If separate EC2 curl reaches instance but hairpin fails • Enable source/destination checking override for the instance? Normally not required, but check: ◦ In EC2 console select the LiveKit instance → Networking → Change Source/Dest. Check = Disabled? (should be enabled normally) — report current value. • Ensure NLB has “Preserve client IP” not needed for TCP, but hairpin behavior can be impacted by rp_filter; you showed rp_filter = 2 (loose mode) which is fine. 5) Short-term workaround (if hairpin cannot be enabled quickly) • Point your internal DNS used by VPN clients to the EC2 instance’s private IP (10.148.214.14) instead of the NLB, so clients connect directly inside the VPC and bypass hairpin. This avoids NLB hairpin problems while you debug. If you want, I can give the exact DNS record change to request from your DNS team. 6) Corporate proxy behavior (VPN client saw corporate proxy 502) • From the VPN client run: curl -vk --noproxy '*' https://63.178.59.195/ If this returns the same 502 proxy page, the client path is still being intercepted; ask your network team to allow direct CONNECT/forward to the NLB IP or add the domain to the proxy allowlist. I can draft a short allowlist request text if you want. What I need from you now (pick one and paste the output) • Run step 1 and tell me: NLB Scheme and Cross-zone status. OR • Do step 2 (curl from another EC2 + tcpdump on LiveKit host) and paste the tcpdump + curl outputs. I’ll interpret the result and give the single next change to make (NLB attribute, TG change, DNS tweak, or proxy allowlist).
f
1. it is internal scheme and Cross-zone load balancing is on
t
Thanks — good to know NLB is internal and cross-zone is enabled. Next step: confirm whether the NLB → instance path works for other VPC hosts (this will tell us if the issue is hairpinning vs. general NLB routing). Please run one of these tests and paste the outputs. Test A — from another EC2 in same VPC (preferred) 1. On your LiveKit instance, start capture: sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 40 1. From a different EC2 in the same VPC/subnet, run: curl -v http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' 1. Paste: ◦ The curl output (from the other EC2) ◦ The tcpdump lines captured on the LiveKit instance Interpretation: • If tcpdump shows inbound SYN/SYN-ACK and the curl returns a response, NLB→instance path is OK and the remaining problem is hairpin or client proxy — we’ll give DNS or proxy workarounds. • If tcpdump shows nothing or only outbound SYNs from the LiveKit host (no inbound replies), NLB is not forwarding traffic to the instance — we’ll inspect TG ARN, subnet/AZ mapping, and ENI attachments next. If you cannot start another EC2 quickly, run this variant from the LiveKit host (hairpin test): 1. On LiveKit host start tcpdump: sudo tcpdump -n -i any host 63.178.59.195 and tcp port 80 -c 40 1. In a second shell on same host: curl -v --noproxy '*' http://63.178.59.195:80/ 2>&1 | sed -n '1,120p' 1. Paste both outputs. After you paste results I will tell you the single next AWS console command to run (TG describe or ENI check) and the exact change to make.
Open in Slack
PreviousNext
Powered by