https://pinot.apache.org/ logo
n

Neil Teng

04/20/2021, 5:23 PM
Hi some of my pinot instances are dead for unknown reasons. Is that any way to recover that? Because queries may fail if it is directed to a dead broker.
I am using the pinot 0.7 docker
m

Mayank

04/20/2021, 5:26 PM
You can restart the nodes to bring them back
However, please also check the logs to see why they went into this state (could be OOM).
n

Neil Teng

04/20/2021, 5:33 PM
docker restart pinot- controller/broker/server does not recover the state.
BTW, where is the log?
I can see in the cluster manager UI that has a error under liveinstance config:
Copy code
{
  "code": 404,
  "error": "ZKPath /PinotCluster/LIVEINSTANCES/Controller_172.18.0.7_9000 does not exist:"
}
m

Mayank

04/20/2021, 5:44 PM
Logs should be on the instances themselves (whereever the log4j was configured)
The restart should show some info in the logs on why this is happening.
n

Neil Teng

04/20/2021, 6:09 PM
I tailed the log from where it starts to throw exceptions
@Mayank any idea of the error?
m

Mayank

04/21/2021, 2:36 PM
Will take a look shortly