https://pinot.apache.org/ logo
#getting-started
Title
# getting-started
t

Tiger Zhao

09/10/2021, 8:24 PM
Does Pinot automatically delete the generated indices from the servers after deleting a segment? I'm running into an issue where I would delete the segment through the REST API, but it leaves the index files under PinotServer/index behind. The indices built up over time from my test tables and now the servers are out of disk space
j

Jackie

09/10/2021, 8:32 PM
The indices should be in the same folder as the segment, and removed along with the segment. Can you please check what files are left over on the servers?
t

Tiger Zhao

09/10/2021, 8:42 PM
It seems that all the segments are still remaining in PinotServer/index. I think my cluster somehow ended up in a weird state where it shows something like 1000/0 segments in the controller UI for that table.
It doesn't show the segments anymore through the UI but still returns the segments when viewing them through the REST API. Trying to delete through the REST API again seems to do nothing now.
j

Jackie

09/10/2021, 9:31 PM
Can you please check the idealstate and external view of the table?
t

Tiger Zhao

09/10/2021, 9:39 PM
The ideal state is empty. The external view actually shows all of the segments as OFFLINE
j

Jackie

09/10/2021, 10:40 PM
The external view still exist?
Can you check if there is any ERROR in the server log?
t

Tiger Zhao

09/12/2021, 6:01 PM
Yeah it looks like the external view still exists. I don't see any errors in the server logs. I think this issue happened when I tried to reset all of the segments but it timed out. After the failed reset, it left all of the segments OFFLINE. I then tried to delete them and now it is in the current state.
j

Jackie

09/13/2021, 5:41 PM
Can you search if there are state transition from OFFLINE to DROPPED logged on the server side? If not, then it might be controller/zookeeper issue
t

Tiger Zhao

09/13/2021, 8:46 PM
Ah I do see a bunch of transitions from OFFLINE->DROPPED. What does this mean in terms of the current state of the cluster?
j

Jackie

09/13/2021, 9:30 PM
Which
current state
were you referring to?
Basically this state transition is trying to delete the segment from the server local storage
ONLINE -> OFFLINE
is offloading the segment from the memory, and
OFFLINE -> DROPPED
is deleting the segment file
t

Tiger Zhao

09/14/2021, 2:01 PM
I just meant the state of that table in general. It's been almost a week and it looks like the segment files are still there. I think the process of deleting the segment files got stuck? Is there a recommended way of cleaning up the segments/table?
I just deleted the entire table and that seems to have cleaned up the segments.