I think the rebalance API does not work as expecte...
# generalr
Rohit Yadav
11/30/2023, 6:48 PMI think the rebalance API does not work as expected for a very large table(34TB). I ran into a dead host situation, where I removed the tenant tag from it and disabled it. I then tried to rebalance the servers with reassign instances flag. The response does not list the dead instance but the Ideal state is not removing the dead instance in the segment assignment config.
What are my options to rebalance the table? I do not want to try the rebalance with downtime as I suspect there will be missing deep store copy for some segments.
j
Johan Adami
12/01/2023, 4:01 AMrebalance without downtime happens in several iterations. it won’t always remove that server right away.
r
Rohit Yadav
12/01/2023, 5:16 AMWould it guarantee atleast one replica availability? In the past we have faced issues where we lost 10% of our segments with downtime rebalance.
Rohit Yadav
12/01/2023, 10:28 AMSo we tried some thing hacky but it worked well. We copied the dry run output of rebalance and manually updated the ideal state as rebalance API was not doing it(will revisit the code). The diff was exactly the dead instance being replaced with multiple live instances and it did not even require a rebalance after updating the ideal state.
j
Johan Adami
12/01/2023, 4:02 PMas long as nothing else moved around, that’s a good workaround. the rebalancer does the same thing except it waits for idealstate == externalview before proceeding
Johan Adami
12/01/2023, 4:03 PMrebalancing has a “min replicas” option that we always set to
num replicas -1r
Rohit Yadav
12/06/2023, 9:12 AMGot it, thanks for these details, I will try
min replicas