hey friends... I was issuing rolling updates for ...
# troubleshooting
l
hey friends... I was issuing rolling updates for pinot-servers with
kubectl
however I noticed that when I run this command I always get a brand new server and have to issue rebalances again, is restarting servers something that requires rebalancing? I'm pretty sure it must be something funky going on with our config
d
How have you deployed the cluster in the first place?
l
I’m using GKE, with the helm chart provided in the OSS
d
and what command do you use to update your servers?
l
what do you mean to update my severs? to restart them you mean?
i just do
kubectl rollout restart statefulset/pinot-server
d
I was issuing rolling updates for pinot-servers with kubectl
Could you provide the specific of what you did here.
and could you describe what you mean by:
I always get a brand new server
l
when i do that then i get 2 new servers and things start to look like this:
d
Ok
Check if you have
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
in your pinot server configmap
l
lemme check
d
you would need that property for all components
With this, your pods will keep the same identity in Helix.
l
all components need that yes?
pinot.set.instance.id.to.hostname
d
servers for sure
l
oh yes i def don’t see that being set for the server
d
let me check other nodes
Yes, they all need it. controller, minion, server and broker
l
yep they all have it in my case except for the server
so what does this do in particular?
d
Tell Helix that the cluster member should use its OS hostname as an identifier rather than IP/PORT.
In a k8s statefulset, IP is dynamic, hostname is preditable per pod instance
l
oh i have different environments and in sandbox i do see the name being different
like this
d
Making it a default value in the helm chart would be a sensible thing to do. If you want to contribute to the OSS helm chart, that is an easy PR.
l
message has been deleted
i think that was by default what the helm chart did i feel like we change the configs but i will double check
now it makes sense cause i remember when i used to do rolling updates in sandbox i wouldn’t have to do any of the things i was doing now in the new env lol
rolling restarts* sorry
thank you so much for your help
d
You’re welcome 👍