https://pinot.apache.org/ logo
Join Slack
Powered by
I am getting the following error when trying to dr...
# troubleshooting
y
I am getting the following error when trying to drop a test broker.
Copy code
{
  "code": 409,
  "error": "Failed to drop instance Broker_172.17.0.2_8099 - Instance Broker_172.17.0.2_8099 exists in ideal state for brokerResource"
}
How do i remove the instance from ideal states and also drop it ?
n
i’m guessing you already untagged the broker before trying to drop?
y
I am not really sure. We just killed the docker instance.
what do you mean by untag. We are currently just using default tenants.
n
what are you exactly doing when you say 
trying to drop a test broker
? simply shutting down the broker?
If you’re trying to scale down your brokers - By untag, I mean removing the “DefaultTenant” tag on that broker, so that the ideal state removes that broker. You can do that with
Copy code
@PUT
  @Path("/instances/{instanceName}/updateTags")
y
Okay. How do we include it as part our kubernetes deployment ? we want to simply remove the broker if it killed ?
n
firstly, how did you get that error? did you invoke this on the broker
Copy code
@DELETE
  @Path("/instances/{instanceName}")
y
Yes.
n
got it
y
Is there a config that allows us to auto delete / untag dead brokers ?
n
okay, so as of now, it will have to be 2 step. Untag, then drop.
we could look into making dthe drop call also untag before dropping
there is no auto untag/delete
y
Okay. Also how long goes it take to untag. I have untagged it. but still it is taking time to delete.
n
can you check in zooinspector if it got successfully untagged?
y
Copy code
{
  "id": "Broker_10.45.203.83_8099",
  "simpleFields": {
    "HELIX_ENABLED": "false",
    "HELIX_ENABLED_TIMESTAMP": "1600191127639",
    "HELIX_HOST": "Broker_10.45.203.83",
    "HELIX_PORT": "8099"
  },
  "mapFields": {},
  "listFields": {
    "TAG_LIST": []
  }
}
Copy code
"airlineStats_OFFLINE": {
      "Broker_10.59.100.46_8099": "ONLINE",
      "Broker_10.45.203.208_8099": "ONLINE",
      "Broker_10.45.203.83_8099": "ONLINE",
      "Broker_10.59.100.48_8099": "ONLINE",
      "Broker_10.59.100.47_8099": "ONLINE"
    },
but this server is still in IDEALSTATES
n
i see. let me confirm with someone that untagging is all thats needed for idealstate update
@Mayank @Xiang Fu can you take a quick look? untagging a broker is all that should be needed for this right?
x
untagged the broker and rebuild the brokerResources idealstates?
n
try this
Copy code
@POST
  @Produces(MediaType.APPLICATION_JSON)
  @Path("/tables/{tableName}/rebuildBrokerResourceFromHelixTags")
m
@Yash Agarwal Going through the thread, one question: Is the broker permanently dead in your case, or is it a transient error?
For example, if it is expected to come back online at some point, then no action is required. It will become offline in external view once it is dead.
y
permanently dead. We are testing an auto scale up/down strategy for our brokers.
m
I see
n
hmm. so it is 3 step to just scale down.. 1. untag 2. rebuild 3. drop it would be nice to have a one shot operation for this
y
Also step 2 needs to repeat for all the tables.
n
true. Let me start with documenting this. then i’ll create an issue to make it easier
Open in Slack
PreviousNext
Powered by