Hi, we're seeing high disk usage on the minions fr...
# troubleshootingt
Tiger Zhao
09/16/2022, 3:32 PMHi, we're seeing high disk usage on the minions from the ReatimeToOfflineSegments task. Is it expected that pinot will keep the segments on disk after finishing the task? Also we noticed that when running
lsofm
Mayank
09/16/2022, 3:33 PM
@Xiaobing ^^
x
Xiaobing
09/16/2022, 4:16 PMhi Tiger, thanks for debugging this. The task shouldn’t keep any input segments or newly generated segments on disk after completion. If so, that’d be a bug.
could you help open an issue about this, and your findings with lsof. Thanks!
Xiaobing
09/16/2022, 4:17 PMPinot is keeping a large number of deleted segments open stillperhaps some cleanup code is not put in a finally{} block
t
Tiger Zhao
09/16/2022, 6:34 PMmade an issue: https://github.com/apache/pinot/issues/9415
Tiger Zhao
09/19/2022, 2:06 PM@Xiaobing if the bucket time period for the ReatimeToOfflineSegments task is set to one day, would pinot try to download that entire day of segments before running the task? Or would it be done in smaller batches?
x
Xiaobing
09/19/2022, 4:30 PMiirc, the current implementation would download all segments before processing them
t
Tiger Zhao
09/19/2022, 6:36 PMI see, and as the offline segments are created, are those (and the realtime ones) immediately deleted off the minion disk or would that deletion occur after all the data has been processed for that day?
x
Xiaobing
09/19/2022, 10:47 PMit’s deleted after segment got generated and compressed (link to code)
and did you see logs like in your case?
Copy code
LOGGER.warn("Failed to delete input segment: {}", inputSegmentDir.getAbsolutePath());t
Tiger Zhao
09/22/2022, 8:22 PMhaven't seen any log messages like that. But given that it downloads all the segments, I'm thinking it's possible the disk usage is just coming from that