https://pinot.apache.org/ logo
Join Slack
Powered by
<@UGRJA9TEH> We are having pinot cluster setup whe...
# general
a
@Xiang Fu We are having pinot cluster setup where we are have enabled HPA (Horizontal Pod Autoscaling) where minimum - 3 server, maximum - 6 servers. There was lot of traffic due to which the server reached to it's maximum pods and when the traffic was minimised the HPA reduce the number of server to 3 which I am able to see on the cluster but when checking the pinot-controller it is still showing other 3 server as status - 'Dead' and some of the table segments are showing 0 server address in their details. Can you please help us. We restarted all the server and controller but still the status remains same -
x
Added servers won’t be auto delete.
To remove them you need to mark those instance to other tenant name, then rebalance all tables, then delete pinot instances using controller API.
a
It was removed automatically I can see in the kubernetes it is showing total 3 server
but in the controller it is showing total 6 server with 3 server status as 'DEAD'
@Xiang Fu since in the controller it is showing more than 3 server. After reloading the table segments it is considering the those 6 servers and for some segments it is waiting for the response from the dead server
For few segments it is showing 'Updating' for long time now
x
yes, you need to update the server tenant tag then rebalance
otherwise controller will think those servers are still valid, but just offline
Since minion is a stateless svc, there is an implementation for auto deletion after a certain time to delete the instances. But broker/server side auto scaling is not yet a priority
It requires a series of manual steps
a
Thank you @Xiang Fu will try the above steps
2 Views
Open in Slack
PreviousNext
Powered by