https://pinot.apache.org/ logo
Join Slack
Powered by
hey friends, I have a question, in our dev pinot c...
# troubleshooting
l
hey friends, I have a question, in our dev pinot cluster (built on top of k8s), we have been scaling up and down things like servers/controllers and all, once we scale down something we can see in the pinot UI that servers that we scale down from go to Dead status, when we add a REALTIME table at that point, for some reason the status is Bad, we have fixed this by removing those servers and then eventually we can add this, any reason why Realtime tables complain even tho those servers are now gone? do we have to run a rebalance before we even add a new realtime table?
m
Perhaps you or used the same tag as the dead servers?
l
that’s based on this basically, yes?
Copy code
"tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant",
    "tagOverrideConfig": {
      "realtimeCompleted": "DefaultTenant_OFFLINE"
    }
  },
n
If you've taken that server down, you need to also untag it and then drop instance https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-servers
l
sorry how do i drop the instance? I think I got the gist
so the table is using all the servers that have the same tag to distribute the data, so what i have to do is untag the dead servers and i should be able to create the table afterwards yes?
this is it yes
why do i need to untag at all? can i just not drop the instance?
n
i think you won’t be able to dropInstance, if the instance has a tag that is being used by some table
but give it a shot
l
it did let me drop instance
i guess that since the table wasn’t yet in production
it did let me
as in we haven’t deployed the table*
when we deployed the table we god status bad and we deleted it
deleted the instances and deployed table again and it worked
but if table was there then we had to untag, rebalance and delete yes?
n
Yes that's right
Open in Slack
PreviousNext
Powered by