Hi, I am running datahub-upgrade with SystemUpdate parameter and getting below exception. It says index not found. Ideally the index should get created if it doesn't exist

Caused by: org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [vdc_dev_containerindex_v2_1702876349208]]
	at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:209)
	at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)
	at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)
	at org.opensearch.client.IndicesClient.delete(IndicesClient.java:118)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.reindex(ESIndexBuilder.java:358)
	at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.buildIndex(ESIndexBuilder.java:225)
	... 22 common frames omitted
	Suppressed: org.opensearch.client.ResponseException: method [DELETE], host [<https://c9c1211d1a9049b0b64be0193936cdb7.ecenp.visa.com:9243>], URI [/vdc_dev_containerindex_v2_1702876349208?master_timeout=30s&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true&ignore_throttled=false&timeout=30s], status line [HTTP/1.1 404 Not Found]
Warnings: [[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices.]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [vdc_dev_containerindex_v2_1702876349208]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"vdc_dev_containerindex_v2_1702876349208","index":"vdc_dev_containerindex_v2_1702876349208"}],"type":"index_not_found_exception","reason":"no such index [vdc_dev_containerindex_v2_1702876349208]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"vdc_dev_containerindex_v2_1702876349208","index":"vdc_dev_containerindex_v2_1702876349208"},"status":404}
		at org.opensearch.client.RestClient.convertResponse(RestClient.java:375)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:345)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
		at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)
		... 27 common frames omitted

Could there be another process running modifying your index state simultaneously? We do have a check that runs immediately before this code executes that verifies the index exists, so not sure how it would reach this path with the index not existing. See here. Also what version are you running?

@orange-night-91387 I am using Datahub v0.12.0. and Java 17 When I tried using Java 11 it worked on my laptop. But all our Environment nodes are on Java 17. No other process is running or modifying the index. Yesterday, I checked the code I also came to the same conclusion that this path would not even come. Could there be a possibility that OpenSearch doesn't work as expected on Java 17?

It's hard to know, we haven't done any testing on Java 17 with the current release. We are working on getting the project up to Java 17 currently though. (PR got merged) If it's possible definitely at least try with Java 11.

For us Java 11 is not possible on our environments due to cybersecurity concerns. But I will try to do a fresh setup on a different node with Java 17 to be 100% sure something else is not causing this issue.