Hello, I have a question about how the controller ...
# troubleshooting
v
Hello, I have a question about how the controller handles rebalancing segments. I am more interested in the following aspect: is there any downtime while moving a segment from a server to another ? I see that in the manual rebalance job you can specify downtime=false only if you have replication. Is the mechanism behind controller rebalancing the same ?
y
Yes, the rebalance API takes in a config
minAvailableReplicas
that is used to determine the number of segments moved at a time. For instance, if you have 3 replicas and
minAvailableReplicas
is set to 1, at most 2 replicas of any segment can be moved at a time. The controller's rebalance algorithm will ensure that the 2 replicas are up before moving the third one for any segment.
v
what if there is only one replica ?
Also if you have a 2 servers, a realtime table, 1 kafka partition and no replication. All segments will be built and reside on a single server unless a rebalance job is run Does the realtime to offline job rebalance the segments ?
y
You can't avoid downtime with a single replica.
Realtime to offline job will move segments from the RT table to the offline table in a hybrid table, so yes the segment assignment there will be completely changed. You can also use
COMPLETED
segment relocation within an RT table (works like a periodic automated rebalance) - https://docs.pinot.apache.org/operators/operating-pinot/instance-assignment.
v
After some testing it indeed seems like the realtime to offline moves the segments with balance in mind I can confirm instance assignment helped with balancing completed segments on the servers tagged with "realtime"
👍 1