Hi Team, we have pinot as persistence layer in ou...
# general
p
Hi Team, we have pinot as persistence layer in our analytics application. what i have observed when we upgrade our application that along with application when pinot pods are also restarted. post restarting tables go in BAD state not the case always. based on my past experience this generally seems to happen as segments go in BAD state as their ideal state and external view state do not match. it seems to me that some segments are in BAD state as during redeployment state transitions got stuck while others segments are in BAD state as sync between states in views is not longer maintained. what is the recommended approach when upgrading. below is what i can think of kindly confirm if this indeed is recommended or point me to the right process. 1> Disable all tables using API to stop querying and ingestions. 2> Commit all current consuming segments using API. 3> Upgrade. 4> Enable state of all tables so new segments start generating and DB is available for querying again. Regards
k
That’s too complicated.. all you need is a rolling restart.. server automatically put themselves in disabled for queries mode during shutdown and enable when it starts back up..
During the rollingrestart call the health check api before going to the next node
p
Hi Kishore, Thank you for your response. i don't get you when we say performing rolling restart and checking api before going to next node. we simply apply an helm upgrade with pinot enable =true which results in pinot getting restart pod by pod and we have minimum control during this process. can you please point me to any blog or doc which provides a overview of the rolling restart process you mention here.
Sorry i get what u mean with rolling restart we had upgraded from OSS pinot 1.1.0 to 1.2.0 by upgrading components 1 by 1 controller --> broker --> server. i rephrase my question a bit. if i do an hemp upgrade of entire application of which pinot is a part all the pods including pinot will be restarted most of the times after restart things are fine while in some cases few tables go into BAD state as segment show difference between external view and ideal state post restart. how can we avoid this provided helm upgrade is not avoidable. can we use pauseConsumption --> commit all consuming segments --> upgrade --> resumeConsumption so this problems of segment states do not occur as having a fluctuating behavior is bad for customer experience. would you recommend this even if it tedious. i have tried rolling restart that process too is quite tedious. or should be avoid restart of pinot during the application upgrade via helm altogether?