Hi team, does anyone have experience migrating from the default instance and segment assignment strategy to the Pool-based instance assignment and replica-group assignment strategy? Want to learn any pain-points or anything that should be cautious from you

@Jackie ^^

we’ve done this several times with no real issues. we typically think of it as just a “rebalance”, so if you want to avoid downtime (set the right corresponding flags: reassignInstances, keep N-1 replicas up, no downtime, etc). And we like to set the minimizeDataMovement flag in our table configs to avoid moving around more segments than is needed

thank you! @Johan Adami One concern I have with rebalance is that even under no downtime mode, if some servers has no extra capacity to hold the extra replicas, the rebalance will actually be affected. How do you solve it or have you seen issues related to this?

unfortunately that is an open feature request, https://github.com/apache/pinot/issues/10002

Ah, didn't find it. Thanks!

this is where

minimizeDataMovement

can help, but I can’t think of a better way other than, make sure you have at least 2x the disk space

right. we now have to ensure the disk space first

or maybe one way is to rebalance onto brand new servers, then take down the original ones

Hi @Johan Adami, want to recap on this topic. I have a question regarding the zookeeper usage of the pool-based instance assignment. Since each table will have a dedicated znode under instancePartitions, will this be a concern for a large cluster? What's the biggest cluster you guys have with the pool-based assignment?

it can be, and we’ve definitely had to increase the size of each zk node as well as increase its heap memory before

why would the rebalance with downtime cause ZK nodes spike? Is it rebalance from no-pool to pool-based?

maybe @David Yang knows better. I just anecdotally have seen an increase in ZK nodes during rebalances

Thank you Johan.

i think we’ve typically seen ZK node spikes due to helix messages propagating around the system for segment state transitions due to the high # of segments we have

@Si Lao We rarely run into issues caused by too many tables. Usually the issues are caused by too many segments in a table. I cannot imagine a cluster with 100K tables

ty jackie. i realize the question was asking about table count and not segment count.

Thank you both! I think helix messaging regarding state transition is not special in terms of pool-based. It also happens for normal balanced assignment. As long as there are massive rebalance happening, these state transition messages can spike.