Hi, we recently had an issue were there were many ...
# troubleshootf
faint-painting-38451
01/11/2023, 6:11 PMHi, we recently had an issue were there were many lingering indices for one of our custom entities that weren't getting removed. Tracked down where it should have been removing these indices to ESIndexBuilder.java > buildIndex(). One thing I noticed in there is that if for whatever reason it fails to delete the old index, that old index will be stuck and will need to be manually deleted since this function only deletes the active index after setting up the new one. What I was thinking was adding a block at the end of the function here that would remove any indices that follow the index name pattern and don't have an alias to prevent indices being leftover. Does that seem like a good solution that would be wanted upstream or is there already something in place to clean up the indices that we would need to add our custom entity to?
a
astonishing-answer-96712
01/11/2023, 6:22 PM@brainy-tent-14503, is this something we might accept?
b
brainy-tent-14503
01/11/2023, 9:39 PMThere is currently no code to clean-up indices that for whatever reason failed a reindex. This is true, I could see a need to identify and clean up non-aliased indices after some retention period. The reindex itself has some changes in a PR however this section around index clean-up remains largely untouched. One of the things that PR does is to actually make the reindex less error prone in the first place. Also note that in the PR the reindexing is not going to be executed as part of GMS but moved to a separate job which runs at install/upgrade times. This new process will actually clone the existing index which also lacks a general clean-up strategy. So in general it would be useful to have a method to identify both orphaned indices and clones for removal as long as it can consider a retention period based on the index created timestamp or the index naming conventions. Happy to discuss further!
brainy-tent-14503
01/11/2023, 9:41 PMDocumentation around the new reindexing is in the works as well.
f
faint-painting-38451
01/12/2023, 3:27 PMCool, I like that it's going to be separate from the GMS, we have had some issues with GMS taking a while to start up due to re-indexing in the past. I could work on the function to identify the orphaned indices if that's not already being done as part of the PR.
faint-painting-38451
01/20/2023, 9:32 PM@brainy-tent-14503 I have the function to retrieve orphaned indices, should I add it to ESIndexBuilder and create a PR to the Elasticsearch PR that's currently active? Also kind of unrelated, but we recently had an issue where it kept reindexing a particular index each time the GMS started despite no changes. What happened was we removed a field from our custom aspect and the reindex part didn't handle that properly. Each time it reindexed, the deleted field was carried over from the previous index. Because of that, the next time it checked for reindexing, it tried to reindex the same index since it saw that it needed to remove the field. We were able to resolve it by manually removing the field from the current index and then it wasn't carried over to the new index on the next reindex. I imagine that removing fields is a rare occurrence and we were able to resolve it manually, but did want to bring attention to the issue we had.
b
brainy-tent-14503
01/20/2023, 11:31 PMYes, please point it to this
feat/elasticsearch-optimization-ext