Hi all, K8s started GMS service and reported the following error：datahub version:0.8.41

06:17:55.194 [pool-13-thread-1] ERROR c.l.d.g.a.service.AnalyticsService:264 - Search query failed: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
06:17:55.199 [pool-13-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler:95 - Unexpected error occurred in scheduled task
java.lang.RuntimeException: Search query failed:
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:265)
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getHighlights(AnalyticsService.java:236)
	at com.linkedin.gms.factory.telemetry.DailyReport.dailyReport(DailyReport.java:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:260)
	... 15 common frames omitted
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<http://10.196.0.102:9200>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahub_usage_event]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"}],"type":"index_not_found_exception","reason":"no such index [datahub_usage_event]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"},"status":404}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
		... 19 common frames omitted

re-run elasticsearch-setup job?

yes, Already initialized

yellow open datahubexecutionrequestindex_v2                          SDlCHuWvR8-eLPXUjbOogA 1 1 0 0   208b   208b
yellow open dataset_datasetprofileaspect_v1                          DCXQvmGPTTad8V6kWgfS2A 1 1 0 0   208b   208b
yellow open datajobindex_v2                                          QFm7yFMATaSY_iyRdj2dcA 1 1 0 0   208b   208b
yellow open datahubsecretindex_v2                                    LvsldXnnSWaPmC4tQq__qQ 1 1 0 0   208b   208b
yellow open mlmodelindex_v2                                          mUIrcO3aQ0i8NDv8vPr8NA 1 1 0 0   208b   208b
yellow open mlmodelgroupindex_v2                                     i2yu3GBwS5Wu6woH6s2asA 1 1 0 0   208b   208b
yellow open dataflowindex_v2                                         e2EuJ33HTCem1GeEy0Yu5g 1 1 0 0   208b   208b
yellow open datahubpolicyindex_v2                                    ppI_FVT_QIec6xT2m2s0rw 1 1 5 0 10.4kb 10.4kb
yellow open assertionindex_v2                                        QDbAdlUrTZmcn_a-UYdIOA 1 1 0 0   208b   208b
yellow open corpuserindex_v2                                         M8Ix6GWCQT-M0k3ErmB2Pg 1 1 0 0   208b   208b
yellow open dataprocessindex_v2                                      Ec3JBFolQsa9piSUNRM6Nw 1 1 0 0   208b   208b
yellow open chartindex_v2                                            GElpEciFSwuDCAGuNSC3eA 1 1 0 0   208b   208b
yellow open tagindex_v2                                              YPWP1kRFQOuCtxl1Yky9-Q 1 1 0 0   208b   208b
yellow open mlmodeldeploymentindex_v2                                x1YM4Em8Qwy6zXPxRMGn3Q 1 1 0 0   208b   208b
yellow open datajob_datahubingestioncheckpointaspect_v1              jNXmCPrkQZ6M7vBQEinvlQ 1 1 0 0   208b   208b
yellow open dataplatforminstanceindex_v2                             A_MQgu70RmWzPZStnLfKMw 1 1 0 0   208b   208b
yellow open dashboardindex_v2                                        EL3UjaQuS4i78pM2d3rIYw 1 1 0 0   208b   208b
yellow open assertion_assertionruneventaspect_v1                     6zbkdikBTOSUOAGrfl6-Xw 1 1 0 0   208b   208b
yellow open datasetindex_v2                                          A_18gErEQOyn7J2joyeHtA 1 1 0 0   208b   208b
yellow open telemetryindex_v2                                        RDTGy3FXRDW93sX09JY9Qw 1 1 0 0   208b   208b
yellow open mlfeatureindex_v2                                        DQ8YxqdTQ2amZLbIujNdJg 1 1 0 0   208b   208b
yellow open dataplatformindex_v2                                     r464x5OcToKWorOmJQ_iZw 1 1 0 0   208b   208b
yellow open datajob_datahubingestionrunsummaryaspect_v1              sGpKdGTZTLGK9-wZoMgk_Q 1 1 0 0   208b   208b
yellow open dashboard_dashboardusagestatisticsaspect_v1              BHIU84ASThaeotSODQvC9w 1 1 0 0   208b   208b
yellow open dataprocessinstanceindex_v2                              hrmxm-9SThq3ZzKajlmbSw 1 1 0 0   208b   208b
yellow open glossarynodeindex_v2                                     u185gpajT1yeBuz89xDf9g 1 1 0 0   208b   208b
yellow open datahubingestionsourceindex_v2                           BTy1qwPIQPi0_IS9RiLNYg 1 1 0 0   208b   208b
yellow open invitetokenindex_v2                                      OKnakTQQR5mH2orYeTHe3Q 1 1 0 0   208b   208b
yellow open datahubretentionindex_v2                                 bJI5bpoNRvO0sUt2MoQT6Q 1 1 0 0   208b   208b
yellow open dataprocessinstance_dataprocessinstanceruneventaspect_v1 Bj4zjd5HS2uNTaUwVEVaFw 1 1 0 0   208b   208b
yellow open system_metadata_service_v1                               Fgx1laspTKWVmkxggmT6fA 1 1 0 0   208b   208b
yellow open dataset_operationaspect_v1                               85A_tfAMRmKdltjEBcEnzA 1 1 0 0   208b   208b
yellow open datahubaccesstokenindex_v2                               ENOfdib_TLuFgc8unlb8Kg 1 1 0 0   208b   208b
yellow open containerindex_v2                                        KdcHBO36RViyKqxt29S9lw 1 1 0 0   208b   208b
yellow open schemafieldindex_v2                                      PmA7p10XSRWAtBn3YIskWQ 1 1 0 0   208b   208b
yellow open domainindex_v2                                           wgCmBQrYTwyCWwvVTav1Sg 1 1 0 0   208b   208b
yellow open testindex_v2                                             MqIuloEZSlybVzlB_3Ptiw 1 1 0 0   208b   208b
yellow open mlfeaturetableindex_v2                                   -MPHqC5TRJKz1Q34lPdJoA 1 1 0 0   208b   208b
yellow open notebookindex_v2                                         0HPoidFvSp6fIdv8HmpvwA 1 1 0 0   208b   208b
yellow open datahubupgradeindex_v2                                   OwXw3SVQTP-Xc0wGGNNG5g 1 1 0 0   208b   208b
yellow open glossarytermindex_v2                                     LWXI0KHmSmClggNPmU61Yw 1 1 0 0   208b   208b
yellow open mlprimarykeyindex_v2                                     t6RANKhOQ8y9RG5nmT3hSQ 1 1 0 0   208b   208b
yellow open corpgroupindex_v2                                        o8wT2OZNRWSIBzPiznRoWg 1 1 0 0   208b   208b
yellow open dataset_datasetusagestatisticsaspect_v1                  MWLSzkBcTNeWuYRHrhEv5g 1 1 0 0   208b   208b

datahub_usage_event points to a hidden index, not sure if it will show up if you /_cat/indices

There are indeed errors in the log

Did the elasticsearch setup job finish successfully ?