https://remoteindian.com/ logo
Join Slack
Powered by
Hello everyone! I am new to Kubernetes and I have ...
# random
s
Hello everyone! I am new to Kubernetes and I have a Kubernetes networking related question. I am adding the details in the thread to avoid cluttering the main channel. EDIT: Stack Overflow question: https://stackoverflow.com/q/72640324/7911552
I am deploying WireGuard on a pod, and I have enabled host networking on this pod (
hostNetwork
). But after few minutes of deploying this pod, the DNS stops working. I have already set the 
dnsPolicy
to 
ClusterFirstWithHostNet
. You can find some more details here https://www.reddit.com/r/kubernetes/comments/vd0g09/dns_stops_working_when_using_hostnetwork/
r
How is your cluster setup?
w
which version of k8s are you using? and are your kube dns pods healthy??
d
I have a feeling that telepresence might solve your underlying problem.
s
@rough-application-24965 the cluster is running on Vultr https://www.vultr.com/docs/kubernetes-at-vultr/
@white-napkin-85004 K8s version is 
v1.23.5
and yes, the kube dns pod is healthy. DNS breaks only on this pod where WireGuard is running. DNS is working fine on other pods.
hey @dry-monkey-93718! I don't understand how a local environment would help me here? I came across telepresence through you message.
r
@silly-thailand-99259 are you following any article / docs?
d
What is it that you're trying to achieve by installing wireguard? People typically reach out for wireguard to connect to k8s services from their local environment.
s
No. I am googling "how to do x" to get this working
Since, i needed to forward a subnet from another node to this node I chose adding 
hostNetwork
privilege as it allows adding adding 
ip route
on the node network.
@dry-monkey-93718 I work on an opensource network management software (OpenWISP). The application needs to reach the devices. I need WireGuard tunnels for that.
šŸ‘ 1
r
is the wiregaurd working when you directly connect to it via IP? how are you saying that DNS isn't working? did the DNS fail when you ran the dig command
are these devices outside the kubernetes cluster? if they are you need to use ingress
s
When I run without host networking ( 
hostNetwork: true
), the DNS works just fine, the WireGuard tunnels come up and I can send traffic through these tunnels from outside. When I re-deploy after enabling host networking, the pods works fine for some time (say 5 minutes), after which I get logged out from the pod (if I have attached a terminal to it using 
kubectl exec
). After this logout, this DNS stops working, The service running on this WireGuard pod cannot resolve any other service FQDN.
are these devices outside the kubernetes cluster?
if they are you need to use ingress
currently I am using NodePort for exposing the WireGuard port. I am aware that I would require an ingress for this, but I want the routing to working before spending time on setting up ingress.
d
Is the pod still running in the cluster? 
kubectl get po
r
why do you need host networking? won't nodeport do the trick?
āž• 1
s
I need to route traffic from another pod to the WireGuard pod for a specific subnet. Then, that traffic is required to be routed through the wireguard interface.
Yes, it is running @dry-monkey-93718
Are you folks familiar with Calico's configuration? I've a feeling that it might be a solution to my problem. But, I don't understand how to configure it for my problem.
r
does vke use calico?
s
yes, it does
r
also how are you routing the traffic from another pod via wiregaurd? are you using egress?
s
In the another pod, I have added an 
ip route
ip route add <subnet> via <private-vpc-ip-of-wireguard-node>
r
curious why are you doing this? if they are in the same kubernetes won't they share the same network as the wiregaurd one?
d
I have a feeling that you're trying to solve problems on k8s like a VM and struggling because it doesn't work like that.
s
I have a feeling that you're trying to solve problems on k8s like a VM and struggling because it doesn't work like that
I agree to this. This is my first time working with K8s. šŸ˜…
if they are in the same kubernetes won't they share the same network as the wiregaurd one?
the WireGuard tunnels have completely different subnet from the VPC of the cluster. the application is only aware of the WireGuard peers IP address. the application sends network packets with WireGuard peer IP address.
e.g. application running on pod 
app
pings the the device with WireGuard address 
172.16.0.2
. the ICMP request packet has to be routed from 
app
pod to the `wireguard`` pod and reverse with the response packet
d
I remember seeing a very old openVPN k8s tutorial to route traffic from the cluster to your local. Let me try to find that. It might help you.
r
can these two subnets talk to each other?
d
can these two subnets talk to each other?
That's what he's trying to do with 
ip route
on the other pod + wireguard tunnel from this one
r
I suggest doing it via calico instead of changing route table of the pod https://projectcalico.docs.tigera.io/networking/workloads-outside-cluster It's tricky with many layers involved
šŸ’Æ 1
s
I banged my head yesterday on this topic, but I don't understand what Calico's documentation asks me to do
r
You are using two subnets for security?
s
I guess so
I want calico to route my traffic on the 
app
pod to the 
wiregaurd
pod for a specific subnet (say 
172.16.0.0/24
)
r
I think the root cause is that when you use host network you get a vm's IP for the pod and since the subnets are different and connection isn't allowed via security groups or equivalent the app pod isn't able to connect to wiregaurd pod via host networking. without host networking it is working as kuberenetes overlay network is taking care of the networking part
s
@rough-application-24965 I will test your hypothesis in a bit. I remember that I was able to ping the wireguard interface on the 
wireguard
pod from the 
app
pod yesterday. I don't remember whether I was able to ping the device's IP address from the 
app
pod.
d
wrt OpenVPN. Nope, I remembered wrong. The one I'd seen https://github.com/pieterlange/kube-openvpn#routing-back-to-the-client assumes you'll send traffic to this pod at various ports. Now I'm convinced that you've to solve it at one of: ā€¢ Overlay network. ā€¢ CNI level. (Maybe read code from something like https://github.com/gravitational/wormhole/) ā€¢ Vultr, if it's possible. Before that, what I'm trying to understand is if you need to do all of this at all. If you have a single application server that needs to connect to your router devices, you can probably run the wireguard container in a sidecar pattern and as far as I know you'll share network between the containers of the same pod.
āž• 1
šŸ’Æ 1
s
thanks you folks for brainstorming with me over this, I will update how we actually end up solving this problem
šŸ‘ 2
d
Also #sig-network in https://slack.k8s.io/ might be a good place to ask.
šŸ‘ 1
s
If you folks are curious to learn about the network management application: OpenWISP: https://openwisp.org I am working on this PR: https://github.com/openwisp/docker-openwisp/pull/226
4 Views
Open in Slack
PreviousNext
Powered by