Getting the following error log in GMS when I acce...
# troubleshoot
c
Getting the following error log in GMS when I access the "Analytics" page in the UI. I had just recreated my DataHub instance and infra (ES and DB). this has to do with someone interacting with GMS before the ES instances have been set up, right?
Copy code
22:12:21.273 [Thread-1167] ERROR c.l.d.g.a.service.AnalyticsService:264 - Search query failed: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
22:12:21.273 [Thread-1167] ERROR c.l.d.g.a.r.GetHighlightsResolver:35 - Failed to retrieve analytics highlights!
java.lang.RuntimeException: Search query failed:
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:265)
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getHighlights(AnalyticsService.java:236)
    at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.getHighlights(GetHighlightsResolver.java:58)
    at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:33)
    at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:24)
    at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:270)
    at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:203)
    at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:60)
    at graphql.execution.Execution.executeOperation(Execution.java:165)
    at graphql.execution.Execution.execute(Execution.java:104)
    at graphql.GraphQL.execute(GraphQL.java:557)
    at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:482)
    at graphql.GraphQL.executeAsync(GraphQL.java:446)
    at graphql.GraphQL.execute(GraphQL.java:377)
    at com.linkedin.datahub.graphql.GraphQLEngine.execute(GraphQLEngine.java:90)
    at com.datahub.graphql.GraphQLController.lambda$postGraphQL$0(GraphQLController.java:94)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:260)
    ... 17 common frames omitted
    Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<http://compass-elasticsearch.us-west-2.prd.fa.tesla.services:80>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"datahub_usage_event","node":"QkcIA9AKTCGOho3ag0da_Q","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.","caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}},"status":400}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 21 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.]
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
    at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
    at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603)
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
    ... 24 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.]
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
    at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
    ... 28 common frames omitted
b
hey Matt! we've seen this issue before when something is off about how your indices in elasticsearch was created. would you mind running the restore indices command to build up indices again to see if that solves your problem? https://datahubproject.io/docs/how/restore-indices/
l
Hi @clever-garden-23538! Gentle reminder to please stick to our Slack Guidelines & post large blocks of code/stack trace in threads; it’s a HUGE help for us to keep track of unaddressed questions across our various support channels! teamwork
c
sorry!