Hi everyone, I have an issue with a k8s deployment...
# troubleshootingd
Dan DC
09/07/2021, 5:47 PMHi everyone, I have an issue with a k8s deployment. Basically controllers are discovered twice: once via headless service and one more time via regular service. The one discovered through the regular services is always reported as "failed" as there is no ZK entry with the FQDN of the service. Is there any way to fix this?
x
Xiang Fu
09/07/2021, 5:52 PM
headless svc is used for internal pod discovery, e.g. pinot-controller-0, pinot-server-2 …
Xiang Fu
09/07/2021, 5:52 PM
zk svc/headless-svc are there but not exposed externally
d
Dan DC
09/07/2021, 5:53 PMI understand but I don't know why Helix is discovering one controller per service. This doesn't happen with the the brokers or any other node type, just the controllers
Dan DC
09/07/2021, 5:55 PMand I don't know whether this will have side effects
Dan DC
09/07/2021, 5:55 PMI'd like to have controllers reported correctly
x
Xiang Fu
09/07/2021, 5:55 PM
from helix side, each controller will register itself
Xiang Fu
09/07/2021, 5:55 PM
the svc side might be the deep store or vip config？
d
Dan DC
09/07/2021, 5:56 PMI'm not sure if I follow you
Dan DC
09/07/2021, 5:57 PMI have deploy the helm chart with the defaults so the FS is the node HD
Dan DC
09/07/2021, 5:58 PMbut I don't know how this could have anything to do with the controllers showing twice
x
Xiang Fu
09/07/2021, 5:58 PM
hmm, where you find controller showing twice
Xiang Fu
09/07/2021, 5:58 PM
in k8s or logs or helix ?
d
Dan DC
09/07/2021, 5:58 PMin the /instances API and therefore in the UI as well
Dan DC
09/07/2021, 5:59 PMHelix manager returns 2 records for the controller as per my first message
Dan DC
09/07/2021, 6:00 PMthere is a single controller POD
x
Xiang Fu
09/07/2021, 6:00 PM
can you paste a screen shot?
Xiang Fu
09/07/2021, 6:00 PM
we will check on that
d
Dan DC
09/07/2021, 6:02 PMlet me see if I can pull a screenshot, I may need to blur some details though.. bear with me
x
Xiang Fu
09/07/2021, 6:07 PM
sure, in general, each pinot pod will register itself, and it’s true for k8s world, fqdn is just its pod name, svc names are externally, so shouldn’t be counted here
d
Dan DC
09/07/2021, 6:16 PMScreenshot 2021-09-07 at 19.09.25.png,Screenshot 2021-09-07 at 19.10.40.png,Screenshot 2021-09-07 at 19.11.07.png
Dan DC
09/07/2021, 6:17 PMI agree, the fact of using kubernetes should not change anything here
Dan DC
09/07/2021, 6:18 PMUnfortunately I'm still too new to the code and I can't find a solution myself
Dan DC
09/07/2021, 6:19 PMlet me know if that helps
x
Xiang Fu
09/07/2021, 7:33 PM
hmm, seems something goes wrong, I’ll check
d
Dan DC
09/07/2021, 7:34 PMcheers!
Dan DC
09/09/2021, 8:56 AMHey Xiang, I think we can ignore this. It seems someone else was messing with my ZK. I've just done a fresh deployment and the cluster looks good! Apologies for the inconvenience
👍 1
x
Xiang Fu
09/09/2021, 8:47 PM
cool, thanks for conforming 😛
👍 1