Hi team, I'm experiencing some issues in pinot

0.10.0

trying to rebalance an offline table after some offline-servers reached

dead

state (and have been replaced with new, healthy nodes), was hoping to get some extra 👀 on it.

Has EV converged with IS now @Scott deRegt?

It has not. Due to the Timeout exception on

rebalance

operation's

waitForExternalViewToConverge

, IS is not getting updated. Therefore, current IS still includes the

dead

nodes, so impossible for EV to converge on it.

What’s the replication?

I have tested with 2 tables, one with

replication

of

2

and the other with

3

.

And both have this issue?

Yes that's correct. Logs above are for table with rep factor of

3

. I also confirmed for this table that the 2

dead

offline servers in the cluster are in different logical instance assignment groups, meaning every segment of this table has at least 2/3 replicated segments available.

The code you pointed to is not for writing the ideal state of rebalance, it is to detect the change in ideal state when rebalance was happening. So IS should have been updated with removed dead nodes.

In dry run, the

dead

nodes do not appear in the new IS returned. Similarly, in controller logs, the

targetAssignment

does not include the

dead

nodes in this log message when running w/ dry run = false.

What’s the current IS and EV, and do they differ (if so, what’s the difference)? Wondering if they differ even before the rebalance (for some reason) and unable to converge which causes timeout even before new rebalanced IS could be applied

The only difference is the IS still contains the

dead

nodes and the EV does not. I do not see any segments in

ERROR

status.

sounds good

Yes, could test with downtime in this case. We do want to have a good handle on how to recover from a lost server node (expecting this to be fairly common occurrence) w/o downtime as we get ready to go live in a production env.

After offline sync, here’s what’s happening: • In 0.10, the rebalance code first waits for existing EV to converge with existing (pre-rebalance) IS to drain any prior changes, before starting rebalance. • In case a node goes down, IS and EV won’t converge and the code times out. In 0.11, this has been addressed as follows: • When rebalancing, choose “downtime” as well as the newly added options of “minimizeDataMovement”. • If the dead servers have been replaced with healthy ones and tagged appropriately before above, this will do the rebalance without downtime (even though you had to choose it).

So in

0.10

, there is no way to recover and re-achieve

good

table status after a lost

server

node w/ no downtime?

There might be a way, let me dm.

Leaving this here for reference, the other best practice / proposed solution by @~Kishore is logically naming servernames consistently such that if there is a hardware failure and a node is replaced, then the new node will have the same logical server name, such that the existing Ideal State remains valid and the External View is able to converge on it.

Hi @Scott deRegt did you find a way to do this ? im facing the same issue now

In our case, upgrading to pinot `0.11.0`+ and doing a table rebalance with

downtime=true

and

minimizeDataMovement=true

worked for us (as mentioned here). Without more information on your cluster setup, I cannot confidently say whether or not that will solve your problem though.