https://remoteindian.com/ logo
#random
Title
# random
s

silly-thailand-99259

06/15/2022, 6:19 PM
Hello everyone! I am new to Kubernetes and I have a Kubernetes networking related question. I am adding the details in the thread to avoid cluttering the main channel. EDIT: Stack Overflow question: https://stackoverflow.com/q/72640324/7911552
I am deploying WireGuard on a pod, and I have enabled host networking on this pod (
hostNetwork
). But after few minutes of deploying this pod, the DNS stops working. I have already set the
dnsPolicy
to
ClusterFirstWithHostNet
. You can find some more details here https://www.reddit.com/r/kubernetes/comments/vd0g09/dns_stops_working_when_using_hostnetwork/
r

rough-application-24965

06/16/2022, 4:17 AM
How is your cluster setup?
w

white-napkin-85004

06/16/2022, 7:01 AM
which version of k8s are you using? and are your kube dns pods healthy??
d

dry-monkey-93718

06/16/2022, 9:21 AM
I have a feeling that telepresence might solve your underlying problem.
s

silly-thailand-99259

06/16/2022, 9:22 AM
@rough-application-24965 the cluster is running on Vultr https://www.vultr.com/docs/kubernetes-at-vultr/
@white-napkin-85004 K8s version is
v1.23.5
and yes, the kube dns pod is healthy. DNS breaks only on this pod where WireGuard is running. DNS is working fine on other pods.
hey @dry-monkey-93718! I don't understand how a local environment would help me here? I came across telepresence through you message.
r

rough-application-24965

06/16/2022, 9:26 AM
@silly-thailand-99259 are you following any article / docs?
d

dry-monkey-93718

06/16/2022, 9:28 AM
What is it that you're trying to achieve by installing wireguard? People typically reach out for wireguard to connect to k8s services from their local environment.
s

silly-thailand-99259

06/16/2022, 9:28 AM
No. I am googling "how to do x" to get this working
Since, i needed to forward a subnet from another node to this node I chose adding
hostNetwork
privilege as it allows adding adding
ip route
on the node network.
@dry-monkey-93718 I work on an opensource network management software (OpenWISP). The application needs to reach the devices. I need WireGuard tunnels for that.
šŸ‘ 1
r

rough-application-24965

06/16/2022, 9:33 AM
is the wiregaurd working when you directly connect to it via IP? how are you saying that DNS isn't working? did the DNS fail when you ran the dig command
are these devices outside the kubernetes cluster? if they are you need to use ingress
s

silly-thailand-99259

06/16/2022, 9:37 AM
When I run without host networking (
hostNetwork: true
), the DNS works just fine, the WireGuard tunnels come up and I can send traffic through these tunnels from outside. When I re-deploy after enabling host networking, the pods works fine for some time (say 5 minutes), after which I get logged out from the pod (if I have attached a terminal to it using
kubectl exec
). After this logout, this DNS stops working, The service running on this WireGuard pod cannot resolve any other service FQDN.
are these devices outside the kubernetes cluster?
if they are you need to use ingress
currently I am using NodePort for exposing the WireGuard port. I am aware that I would require an ingress for this, but I want the routing to working before spending time on setting up ingress.
d

dry-monkey-93718

06/16/2022, 9:41 AM
Is the pod still running in the cluster?
kubectl get po
r

rough-application-24965

06/16/2022, 9:41 AM
why do you need host networking? won't nodeport do the trick?
āž• 1
s

silly-thailand-99259

06/16/2022, 9:42 AM
I need to route traffic from another pod to the WireGuard pod for a specific subnet. Then, that traffic is required to be routed through the wireguard interface.
Yes, it is running @dry-monkey-93718
Are you folks familiar with Calico's configuration? I've a feeling that it might be a solution to my problem. But, I don't understand how to configure it for my problem.
r

rough-application-24965

06/16/2022, 9:45 AM
does vke use calico?
s

silly-thailand-99259

06/16/2022, 9:46 AM
yes, it does
r

rough-application-24965

06/16/2022, 9:46 AM
also how are you routing the traffic from another pod via wiregaurd? are you using egress?
s

silly-thailand-99259

06/16/2022, 9:47 AM
In the another pod, I have added an
ip route
ip route add <subnet> via <private-vpc-ip-of-wireguard-node>
r

rough-application-24965

06/16/2022, 9:50 AM
curious why are you doing this? if they are in the same kubernetes won't they share the same network as the wiregaurd one?
d

dry-monkey-93718

06/16/2022, 9:50 AM
I have a feeling that you're trying to solve problems on k8s like a VM and struggling because it doesn't work like that.
s

silly-thailand-99259

06/16/2022, 9:51 AM
I have a feeling that you're trying to solve problems on k8s like a VM and struggling because it doesn't work like that
I agree to this. This is my first time working with K8s. šŸ˜…
if they are in the same kubernetes won't they share the same network as the wiregaurd one?
the WireGuard tunnels have completely different subnet from the VPC of the cluster. the application is only aware of the WireGuard peers IP address. the application sends network packets with WireGuard peer IP address.
e.g. application running on pod
app
pings the the device with WireGuard address
172.16.0.2
. the ICMP request packet has to be routed from
app
pod to the `wireguard`` pod and reverse with the response packet
d

dry-monkey-93718

06/16/2022, 9:56 AM
I remember seeing a very old openVPN k8s tutorial to route traffic from the cluster to your local. Let me try to find that. It might help you.
r

rough-application-24965

06/16/2022, 9:57 AM
can these two subnets talk to each other?
d

dry-monkey-93718

06/16/2022, 9:57 AM
can these two subnets talk to each other?
That's what he's trying to do with
ip route
on the other pod + wireguard tunnel from this one
r

rough-application-24965

06/16/2022, 9:57 AM
I suggest doing it via calico instead of changing route table of the pod https://projectcalico.docs.tigera.io/networking/workloads-outside-cluster It's tricky with many layers involved
šŸ’Æ 1
s

silly-thailand-99259

06/16/2022, 9:59 AM
I banged my head yesterday on this topic, but I don't understand what Calico's documentation asks me to do
r

rough-application-24965

06/16/2022, 10:00 AM
You are using two subnets for security?
s

silly-thailand-99259

06/16/2022, 10:01 AM
I guess so
I want calico to route my traffic on the
app
pod to the
wiregaurd
pod for a specific subnet (say
172.16.0.0/24
)
r

rough-application-24965

06/16/2022, 10:05 AM
I think the root cause is that when you use host network you get a vm's IP for the pod and since the subnets are different and connection isn't allowed via security groups or equivalent the app pod isn't able to connect to wiregaurd pod via host networking. without host networking it is working as kuberenetes overlay network is taking care of the networking part
s

silly-thailand-99259

06/16/2022, 10:10 AM
@rough-application-24965 I will test your hypothesis in a bit. I remember that I was able to ping the wireguard interface on the
wireguard
pod from the
app
pod yesterday. I don't remember whether I was able to ping the device's IP address from the
app
pod.
d

dry-monkey-93718

06/16/2022, 10:25 AM
wrt OpenVPN. Nope, I remembered wrong. The one I'd seen https://github.com/pieterlange/kube-openvpn#routing-back-to-the-client assumes you'll send traffic to this pod at various ports. Now I'm convinced that you've to solve it at one of: ā€¢ Overlay network. ā€¢ CNI level. (Maybe read code from something like https://github.com/gravitational/wormhole/) ā€¢ Vultr, if it's possible. Before that, what I'm trying to understand is if you need to do all of this at all. If you have a single application server that needs to connect to your router devices, you can probably run the wireguard container in a sidecar pattern and as far as I know you'll share network between the containers of the same pod.
āž• 1
šŸ’Æ 1
s

silly-thailand-99259

06/16/2022, 10:37 AM
thanks you folks for brainstorming with me over this, I will update how we actually end up solving this problem
šŸ‘ 2
d

dry-monkey-93718

06/16/2022, 10:52 AM
Also #sig-network in https://slack.k8s.io/ might be a good place to ask.
šŸ‘ 1
s

silly-thailand-99259

06/16/2022, 10:52 AM
If you folks are curious to learn about the network management application: OpenWISP: https://openwisp.org I am working on this PR: https://github.com/openwisp/docker-openwisp/pull/226
4 Views