Hi, I tried to kick off a rebalance for one of my ...
# troubleshootingw
Will Gan
10/06/2021, 7:49 PMHi, I tried to kick off a rebalance for one of my tables (moving tenant), but afterward I saw that the idealstate wasn't correct. While before I had two replicas for each segment on a set of servers, now each segment only had 1 replica on the same set of servers (not even the new servers I was trying to move them to), with the exception of the most recent segment that got moved and has the correct number of replicas. Does anyone know what the issue might be? FYI this table is being actively queried.
j
Jackie
10/06/2021, 7:56 PMDid you by any chance change the replication of the table?
Jackie
10/06/2021, 7:57 PMYou can try a dry run and see the ideal state generated
w
Will Gan
10/06/2021, 7:58 PMI have since try to change it to get it back to the original state, but not when I originally did the rebalance.
Will Gan
10/06/2021, 7:59 PMHmm I ran the rebalance from the controller UI and when I did it showed me the generated idealstate which was correct along with the "Rebalance in progress (check controller logs for updates)" stuff
Will Gan
10/06/2021, 8:00 PMbut yeah the idealstate in the Zookeeper browser was diff afterwards
j
Jackie
10/06/2021, 9:20 PMThat is definitely unexpected
Jackie
10/06/2021, 9:20 PMCan you try again with dry-run enabled?
Jackie
10/06/2021, 9:21 PMOh, before that, let's make sure you are checking the ideal state instead of the external view?
Jackie
10/06/2021, 9:22 PMIdeal state should always have 2 replicas, while external view might have 1 replica during the transitioning phase
w
Will Gan
10/06/2021, 9:24 PMYup, it's the idealstate. It's weird because
"simpleFields"."REPLICAS"Will Gan
10/06/2021, 9:33 PM@Jackie Here are some screenshots. After the dry run in the first picture I ran normally, showed successfully in progress, then 2nd pic shows current idealstate.
j
Jackie
10/06/2021, 9:40 PMCan you try the rebalance again and see if the new replica is added?
w
Will Gan
10/06/2021, 10:09 PMno change unfortunately. Is it possible that since there are segments with only 1 replica now nothing will work unless I set downtime to true?
j
Jackie
10/06/2021, 10:57 PMCan you please check the controller log and see why the ideal state is not updated? The log should be under the class
TableRebalancerw
Will Gan
10/06/2021, 11:08 PM@Jackie hmm is this the right way to get that
cat pinot-all.log | grep TableRebalancerj
Jackie
10/07/2021, 1:38 AMSync offline. Currently
TableRebalancer