Hey Team, Need some guidance regarding rebalance realtime servers. We are trying some operational test cases in dev environment. We have 4 server cluster , we have one realtime table which is consuming data from kafka topic having 4 partitions. Real time table has replication factor 2. Now lets say as part of test case one of our server goes down(We stopped it for testing purpose) and we are not able to start same server but we can add one additional server to cluster. Table goes in BAD state (External View != IDEAL STATE) How can we rebalance real time table, i.e can we move completed segments to newly added server to make it balanced (i.e change IDEAL state to include new server but drop previous server from IDEAL STATE which is DEAD)? How can we recreate CONSUMING segment ( hope this will be recreated from checkpoint saved as part of replica). Appreciate any suggestions regarding rebalancing REAL TIME servers/tables.

Thanks @Neha Pawar !!. Appreciate you help. Rebalancing worked as per steps you suggested.

hey, my team is sort of going thru the same thing, does this happen because pinot still believes that these servers are still needed and the tables goes into bad state cause it still doesn’t know it shouldn’t care about servers that are dead?

there is no auto-rebalance, it has to be done manually. so when a server goes down, the segmet state becomes OFFLINE in external view. This causes mismatch with ideal state, resulting in table state BAD. But for queries, this should not have any effect, if you have replication. The brokers know that it should route to only the alive servers. you have to then manually get the down server out of rotation and replace with new oes

what does it mean when the table state is BAD is consumption affected for realtimes? cool that makes sense so if say i have 2 servers replication factor 1 and one of them goes down the other one that is kept alive should still be able to serve the data