Hi some of my pinot instances are dead for unknown...
# troubleshootingn
Neil Teng
04/20/2021, 5:23 PMHi some of my pinot instances are dead for unknown reasons. Is that any way to recover that? Because queries may fail if it is directed to a dead broker.
Neil Teng
04/20/2021, 5:24 PMI am using the pinot 0.7 docker
m
Mayank
04/20/2021, 5:26 PM
You can restart the nodes to bring them back
Mayank
04/20/2021, 5:27 PM
However, please also check the logs to see why they went into this state (could be OOM).
n
Neil Teng
04/20/2021, 5:33 PMdocker restart pinot- controller/broker/server does not recover the state.
Neil Teng
04/20/2021, 5:34 PMBTW, where is the log?
Neil Teng
04/20/2021, 5:37 PMI can see in the cluster manager UI that has a error under liveinstance config:
Neil Teng
04/20/2021, 5:37 PM Copy code
{
"code": 404,
"error": "ZKPath /PinotCluster/LIVEINSTANCES/Controller_172.18.0.7_9000 does not exist:"
}m
Mayank
04/20/2021, 5:44 PM
Logs should be on the instances themselves (whereever the log4j was configured)
Mayank
04/20/2021, 5:45 PM
The restart should show some info in the logs on why this is happening.
n
Neil Teng
04/20/2021, 6:09 PMI tailed the log from where it starts to throw exceptions
Neil Teng
04/21/2021, 1:52 PMserver restart log
Neil Teng
04/21/2021, 1:55 PMpinot-broker restart log
Neil Teng
04/21/2021, 2:00 PM@Mayank any idea of the error?
m
Mayank
04/21/2021, 2:36 PM
Will take a look shortly