https://discord.cloudflare.com logo
Join Discord
Powered by
Only one tunnel gets checked by the load balancer ...
# general-help
n
I have a load balancer with two tunnels attached. For some reason, only one tunnel seems to get monitor checks. I think this might also apply to health check traffic, but I need to double check. Traffic is otherwise split roughly equally between the two tunnels, and when there is an outage in that cluster that affects both, the tunnel that doesn't seem to get monitor checks will still register as "healthy". Does anyone have any idea what might be going on here?
confirmed that all the health checks also seem to go to one tunnel
c
Random shot in the dark. Are the tunnels connected to the same DCs? Tunnel traffic typically through the closest location (not officially though). Wondering if the health checks are coming from a DC where the tunnel is connected to
n
They are both connected to the DFW-A PoP, though different colos
I've turned off the health check's right now to reduce the noise a bit and focus on the knobs I have around the monitors
c
There might be some internals of how the tunnel is traffic is being routed but no clue
n
that's what i'm wondering
i've uninstalled and reinstalled that tunnel to no avail
strange, removing the tunnel that gets all the monitor requests still leaves the other tunnel getting none
umm, why are they both one connector?
how did i get myself into this situation?
wait a minute
on worker 1 (the one getting all the traffic), 
cloudflared tunnel info
returns the same id for both tunnels - the id of worker 1's tunnel on worker 2, the correct ids are concerned
wat
i am bamboozled
c
That’s uh funky. Like the connector or the tunnel IDs?
n
from heavy-worker-1
Copy code
➜  ~ cloudflared tunnel info heavy-worker-1                                                       
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01 
➜  ~ cloudflared tunnel info heavy-worker-2                                                       
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01
from heavy-worker-2:
Copy code
➜  ~ cloudflared tunnel info heavy-worker-1
NAME:     heavy-worker-1
ID:       aff69054-<REST>
CREATED:  2023-05-19 22:41:36.32161 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
7b612dfc-<REST> 2024-01-14T00:34:27Z linux_arm64  2024.1.2 104.13.171.136 1xdfw01, 1xdfw05, 2xmci01 
➜  ~ cloudflared tunnel info heavy-worker-2
NAME:     heavy-worker-2
ID:       b7561864-<REST>
CREATED:  2024-01-14 00:49:37.245692 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE                      
d537845f-<REST> 2024-01-14T00:51:00Z linux_arm64  2024.1.2 104.13.171.136 1xdfw06, 1xdfw09, 2xmci01
heavy-worker-1 gets all the monitor traffic
how is this possible
i continue to find novel ways of breaking computers lol
@Cyb3r-Jok3 i legitimately have no idea how i did this lmao
c
lol heavy-worker-1 work seems cursed
n
really is lol
i'm just gonna delete the tunnel and make a new one
c
Remote managed tunnels for the win
n
that did not fix the issue
i am bamboozled
neither did making the tunnels remote
f
Can you try setting the ha connections to 1 and set them each to different regions (1 and 2, not us or global) I’m very curious how you got this to happen
what do the debug logs look like for API requests to/from argotunnels.com from both tunnels?
n
how would one do this?
also, i too am curious about how i got myself into this mess 😂
i'll go check these
Cloudflared tunnel metrics show even rps to each tunnel
weird
but this disagrees with what logs show when i tail journalctl
i'll do more digging, wonder if something is borked on my end
will be double checking my logging
f
tunnel --ha-connections 1 (undocumented), I thought there was a way to force it to region 1 or 2, but i cant seem to find it. the same effect probably from restarting one till its on a seperate region
but given heavy-worker-1 seems to think that both tunnels have the same UUID I think there is a config issue somewhere
how are you starting the tunnels (cmdline and config)?
PreviousNext
Powered by