<!here> We recently added 2 server replicas in our...
# troubleshootinga
Abhijeet Kushe
09/29/2022, 5:47 PM<!here> We recently added 2 server replicas in our pinot cluster on k8s. The table realTimeconfig also has 3 replicas configured.So each segment is present on every pod.After that I made changes to the schema and set the reload segments flag to true.I noticed that the segments of all pods in k8s happens at the same time due to which application was down for 1 hour.We have 652 segments with 1 day flush time.Total records 7143718 with skipUpsert = true.I do know rebalance segment has a feature of Same problem occurs with server pod restarts from argo.Is there a way to do the segment reload in an uptime fashion.(I do know that rebalance has a flag minAvailable replicas) Does reload have that feature
m
Mayank
09/29/2022, 5:58 PM
I didn’t get your setup, what’s the total number of servers?
a
Abhijeet Kushe
09/29/2022, 6:26 PMWe have total 3 server pods now
m
Mayank
09/29/2022, 9:11 PM
The reload in itself will atomically swap the segments, so that shouldn’t cause any downtime. What change did you do in the schema?
a
Abhijeet Kushe
09/29/2022, 9:14 PMi added 2 fields with default values
m
Mayank
09/29/2022, 9:14 PM
Are they derived fields?
a
Abhijeet Kushe
09/29/2022, 9:14 PMNo new fields but with default values
m
Mayank
09/29/2022, 9:15 PM
Hmm, I was asking because I suspected that creating those new columns/indexes might take some time and put pressure on your system.
Mayank
09/29/2022, 9:17 PM
Essentially, what I am saying is that it is not the reload itself, but the computation it may kick in that would have created pressure on your system. A workaround would be to simply do a rolling restart of servers.
a
Abhijeet Kushe
09/29/2022, 9:22 PMFor rolling restart is there an endpoint that can get us a status of the segment load on a pod ?
m
Mayank
09/29/2022, 9:25 PM
Copy code
@GET
@Path("/health/readiness")
@Produces(MediaType.TEXT_PLAIN)
@ApiOperation(value = "Checking server readiness status")
@ApiResponses(value = {
@ApiResponse(code = 200, message = "Server is ready to serve queries"),
@ApiResponse(code = 503, message = "Server is not ready to serve queries")
})a
Abhijeet Kushe
09/29/2022, 9:57 PMThanks this is 0.9.0 or a new endpoint as I don't see in swagger
m
Mayank
09/29/2022, 9:58 PM
This is on the server. Are you looking at server swagger?
👍 2
a
Abhijeet Kushe
09/29/2022, 9:58 PMNo controller
m
Mayank
09/29/2022, 9:59 PM
But seems like it was added back in July, so 0.9.0 might not have it
a
Abhijeet Kushe
09/29/2022, 10:00 PMOk
m
Mayank
09/29/2022, 10:00 PM
You can check to ensure