Hi Team. Is there a way to manually update IDEALS...
# troubleshooting
t
Hi Team. Is there a way to manually update IDEALSTATES? I have a case where when servers are replaced, and I untag the dead servers from the tenant and perform server rebalance, the dead servers are successfully removed from instanceAssignment, but not from segmentAssignment because they still exist in IDEALSTATES (not present in EXTERNALVIEW). As a result, the table remains queryable but it's stuck in UPDATING status as it cannot update segments from the dead servers, and I cannot drop the dead instances.
m
You should not be needing to manually update IDEALSTATE. The rebalance command is expected to do that. Are you sure rebalance ran successfully?
t
I have a scheduled job that monitors the cluster health and rebalances servers or brokers when there are replaced instances. For servers, the job sequence is - update tags -> rebalance servers -> reload segments -> drop dead instance. The job failed, but server rebalance was successful. For rebalance, i hit this endpoint with a post request
Copy code
f"{PINOT_CONTROLLER}/tables/{table}/rebalance?type=REALTIME&includeConsuming=true&downtime=true
This is the output of running rebalance via the UI. I am not pasting the segment assignments because they are thousands. But there are 12 OFFLINE segments in there that belong to the dead servers
Copy code
{
  "jobId": "032d6b28-6a47-4c55-a76a-99a4be2dfaba",
  "status": "NO_OP",
  "description": "Table is already balanced",
  "instanceAssignment": {
    "CONSUMING": {
      "instancePartitionsName": "player_sessions_active_minutes_CONSUMING",
      "partitionToInstancesMap": {
        "0_0": [
          "Server_10.64.73.140_8098",
          "Server_10.64.73.182_8098"
        ]
      }
    }
  },
  "segmentAssignment": {
Currently the only option I have is to delete the table and recreate it. I tried editing the IDEALSTATES json via the UI, and delete the lines with the OFFLINE servers, but it doesn't seem to work, the updated file isn't saved
x
Typically rebalance with the options reassignInstances=true and downtime=true and includeComsuming should update everything
t
That also produced a "Table is already balanced" result. Weirdly enough, 2 days later, the IDEALSTATES was updated (i didn't take any action to do so other than the already mentioned), so now I was able to drop the dead servers
x
OK, then it's possible that the rebalance takes time for processing
How many threads/cpus you give for the controller
t
each controller has 2 CPU and the default 2 threads
x
i see, so it may due to the controller take a while to finish all the rebalance
how many segments total ?
t
a little more than 9000
is there an endpoint that can be called to check rebalance status?
x
you are using downtime rebalance, so Idealstate is set to the final one, I think the issue here is the helix transistions converge, which requires you to check the ExternalView and IdealState convergence
also each server may need to download segments due to the rebalance