Hi everyone, We are running Pinot with 3 servers on separate nodes and each realtime table segment has 3 replicas. Each night we shut down the VMs in a dev cluster to look for stability issues. This sometimes causes segments to bad. In the docs the instructions are to reset the erroneous segments for a table using the API. This seems to work. Is there a job/task that does this automatically that we have failed to configure or do we need to handle it "manually" through the API?

AFAIK there isn’t a tool to automatically fix this, but I think it should be possible to create a script to automate it. The Pinot UI uses a couple of APIs to figure out if things are working: •

/tables/{tableName}/externalview

- this is the state of segments for a given table •

/tables/${name}/idealstate

- this is the state that it’s trying to achieve. If the results from these APIs doesn’t match then we end up with the

bad

status. So I think if you computed a diff between the output of these two endpoints you would get a list of bad segments and you could then rest them. Do you know why the segments go bad? Is it because the IPs of the VMs are sometimes different after you restart things?

Can you check the debug api (or server log) for segment names when servers are restarted? The expectation is for segments to come ONLINE

Today I just check manually in the Cluster Manager if any segments are faulty and then I curl in a reset for that table and segment. I was thinking that maybe we could just have a rolling schedule of some job that resets all segments for a table for a specific targetInstance even if there are no broken segments or does that disrupt the service too much? (curl -X POST "http://pinot-controller:9000/segments/someTable_REALTIME/reset?targetInstance=serverX.yada.yada" )

What I am a saying is that just shutting down the servers and then restarting them again is not expected to put segments in error state. So if you are seeing that, it is pointing to some other underlying root cause, it would be good to investigate that.

Aha, ok, thanks, I will investigate that the next time it happens.