https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
l

Luis Fernandez

03/28/2022, 5:28 PM
hey friends... I was issuing rolling updates for pinot-servers with
kubectl
however I noticed that when I run this command I always get a brand new server and have to issue rebalances again, is restarting servers something that requires rebalancing? I'm pretty sure it must be something funky going on with our config
d

Daniel Lavoie

03/28/2022, 5:30 PM
How have you deployed the cluster in the first place?
l

Luis Fernandez

03/28/2022, 5:33 PM
I’m using GKE, with the helm chart provided in the OSS
d

Daniel Lavoie

03/28/2022, 5:34 PM
and what command do you use to update your servers?
l

Luis Fernandez

03/28/2022, 5:35 PM
what do you mean to update my severs? to restart them you mean?
i just do
kubectl rollout restart statefulset/pinot-server
d

Daniel Lavoie

03/28/2022, 5:35 PM
I was issuing rolling updates for pinot-servers with kubectl
Could you provide the specific of what you did here.
and could you describe what you mean by:
I always get a brand new server
l

Luis Fernandez

03/28/2022, 5:37 PM
when i do that then i get 2 new servers and things start to look like this:
d

Daniel Lavoie

03/28/2022, 5:39 PM
Ok
Check if you have
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
in your pinot server configmap
l

Luis Fernandez

03/28/2022, 5:39 PM
lemme check
d

Daniel Lavoie

03/28/2022, 5:39 PM
you would need that property for all components
With this, your pods will keep the same identity in Helix.
l

Luis Fernandez

03/28/2022, 5:41 PM
all components need that yes?
pinot.set.instance.id.to.hostname
d

Daniel Lavoie

03/28/2022, 5:43 PM
servers for sure
l

Luis Fernandez

03/28/2022, 5:43 PM
oh yes i def don’t see that being set for the server
d

Daniel Lavoie

03/28/2022, 5:43 PM
let me check other nodes
Yes, they all need it. controller, minion, server and broker
l

Luis Fernandez

03/28/2022, 5:43 PM
yep they all have it in my case except for the server
so what does this do in particular?
d

Daniel Lavoie

03/28/2022, 5:45 PM
Tell Helix that the cluster member should use its OS hostname as an identifier rather than IP/PORT.
In a k8s statefulset, IP is dynamic, hostname is preditable per pod instance
l

Luis Fernandez

03/28/2022, 5:45 PM
oh i have different environments and in sandbox i do see the name being different
like this
d

Daniel Lavoie

03/28/2022, 5:45 PM
Making it a default value in the helm chart would be a sensible thing to do. If you want to contribute to the OSS helm chart, that is an easy PR.
l

Luis Fernandez

03/28/2022, 5:46 PM
message has been deleted
i think that was by default what the helm chart did i feel like we change the configs but i will double check
now it makes sense cause i remember when i used to do rolling updates in sandbox i wouldn’t have to do any of the things i was doing now in the new env lol
rolling restarts* sorry
thank you so much for your help
d

Daniel Lavoie

03/28/2022, 5:49 PM
You’re welcome 👍