https://pinot.apache.org/ logo
Channels
Powered by
y

Yash Agarwal

09/15/2020, 1:44 PM
I am getting the following error when trying to drop a test broker.
Copy code
{
  "code": 409,
  "error": "Failed to drop instance Broker_172.17.0.2_8099 - Instance Broker_172.17.0.2_8099 exists in ideal state for brokerResource"
}
How do i remove the instance from ideal states and also drop it ?
n

Neha Pawar

09/15/2020, 4:37 PM
i’m guessing you already untagged the broker before trying to drop?
y

Yash Agarwal

09/15/2020, 5:13 PM
I am not really sure. We just killed the docker instance.
what do you mean by untag. We are currently just using default tenants.
n

Neha Pawar

09/15/2020, 5:16 PM
what are you exactly doing when you say 
trying to drop a test broker
? simply shutting down the broker?
If you’re trying to scale down your brokers - By untag, I mean removing the “DefaultTenant” tag on that broker, so that the ideal state removes that broker. You can do that with
Copy code
@PUT
  @Path("/instances/{instanceName}/updateTags")
y

Yash Agarwal

09/15/2020, 5:20 PM
Okay. How do we include it as part our kubernetes deployment ? we want to simply remove the broker if it killed ?
n

Neha Pawar

09/15/2020, 5:24 PM
firstly, how did you get that error? did you invoke this on the broker
Copy code
@DELETE
  @Path("/instances/{instanceName}")
y

Yash Agarwal

09/15/2020, 5:25 PM
Yes.
n

Neha Pawar

09/15/2020, 5:26 PM
got it
y

Yash Agarwal

09/15/2020, 5:26 PM
Is there a config that allows us to auto delete / untag dead brokers ?
n

Neha Pawar

09/15/2020, 5:27 PM
okay, so as of now, it will have to be 2 step. Untag, then drop.
we could look into making dthe drop call also untag before dropping
there is no auto untag/delete
y

Yash Agarwal

09/15/2020, 5:33 PM
Okay. Also how long goes it take to untag. I have untagged it. but still it is taking time to delete.
n

Neha Pawar

09/15/2020, 5:40 PM
can you check in zooinspector if it got successfully untagged?
y

Yash Agarwal

09/15/2020, 5:40 PM
Copy code
{
  "id": "Broker_10.45.203.83_8099",
  "simpleFields": {
    "HELIX_ENABLED": "false",
    "HELIX_ENABLED_TIMESTAMP": "1600191127639",
    "HELIX_HOST": "Broker_10.45.203.83",
    "HELIX_PORT": "8099"
  },
  "mapFields": {},
  "listFields": {
    "TAG_LIST": []
  }
}
Copy code
"airlineStats_OFFLINE": {
      "Broker_10.59.100.46_8099": "ONLINE",
      "Broker_10.45.203.208_8099": "ONLINE",
      "Broker_10.45.203.83_8099": "ONLINE",
      "Broker_10.59.100.48_8099": "ONLINE",
      "Broker_10.59.100.47_8099": "ONLINE"
    },
but this server is still in IDEALSTATES
n

Neha Pawar

09/15/2020, 5:42 PM
i see. let me confirm with someone that untagging is all thats needed for idealstate update
@Mayank @Xiang Fu can you take a quick look? untagging a broker is all that should be needed for this right?
x

Xiang Fu

09/15/2020, 5:50 PM
untagged the broker and rebuild the brokerResources idealstates?
n

Neha Pawar

09/15/2020, 5:51 PM
try this
Copy code
@POST
  @Produces(MediaType.APPLICATION_JSON)
  @Path("/tables/{tableName}/rebuildBrokerResourceFromHelixTags")
m

Mayank

09/15/2020, 5:52 PM
@Yash Agarwal Going through the thread, one question: Is the broker permanently dead in your case, or is it a transient error?
For example, if it is expected to come back online at some point, then no action is required. It will become offline in external view once it is dead.
y

Yash Agarwal

09/15/2020, 5:52 PM
permanently dead. We are testing an auto scale up/down strategy for our brokers.
m

Mayank

09/15/2020, 5:52 PM
I see
n

Neha Pawar

09/15/2020, 5:53 PM
hmm. so it is 3 step to just scale down.. 1. untag 2. rebuild 3. drop it would be nice to have a one shot operation for this
y

Yash Agarwal

09/15/2020, 5:54 PM
Also step 2 needs to repeat for all the tables.
n

Neha Pawar

09/15/2020, 5:56 PM
true. Let me start with documenting this. then i’ll create an issue to make it easier
Powered by