The soft references should be from the cached inverted indexes in

BitmapInvertedIndexReader

. Even though the content of the index is on off-heap memory, not sure if keeping all the references would cause memory issue if there are too many inverted indexes