https://datahubproject.io logo
Join Slack
Powered by
Hello! After i started to use latest version of da...
# troubleshoot
c
Hello! After i started to use latest version of datahub, analytics page started to give error in UI called “An unknown error occurred.” on top of the page and all charts are empty now. I checked the logs of gms pod from k8s, i saw an error. The full error is like below:
Copy code
09:57:26.739 [Thread-6973] ERROR c.l.d.g.a.service.AnalyticsService - Search query failed: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
09:57:26.739 [Thread-6973] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler - Failed to execute DataFetcher
java.lang.RuntimeException: Search query failed:
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:245)
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getHighlights(AnalyticsService.java:216)
	at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.getHighlights(GetHighlightsResolver.java:50)
	at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:29)
	at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:19)
	at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:270)
	at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:203)
	at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:60)
	at graphql.execution.Execution.executeOperation(Execution.java:165)
	at graphql.execution.Execution.execute(Execution.java:104)
	at graphql.GraphQL.execute(GraphQL.java:557)
	at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:482)
	at graphql.GraphQL.executeAsync(GraphQL.java:446)
	at graphql.GraphQL.execute(GraphQL.java:377)
	at com.linkedin.datahub.graphql.GraphQLEngine.execute(GraphQLEngine.java:88)
	at com.datahub.metadata.graphql.GraphQLController.lambda$postGraphQL$0(GraphQLController.java:82)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:240)
	... 17 common frames omitted
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<https://vpc-datahub-o67waaz2xr5zttbor35tgmlksa.us-east-1.es.amazonaws.com:443>], URI [/datahub_datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"datahub_datahub_usage_event","node":"M5OibEC5ThKefEm2b1wR4Q","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.","caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
		... 21 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
	at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603)
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
	... 24 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
	... 28 common frames omitted
09:57:26.740 [Thread-6973] ERROR c.d.m.graphql.GraphQLController - Errors while executing graphQL query: "query getHighlights {\n  getHighlights {\n    value\n    title\n    body\n    __typename\n  }\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[getHighlights], extensions={code=500, classification=DataFetchingException}}], data=null}, errors: [DataHubGraphQLError{path=[getHighlights], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
how do i resolve the issue?
e
@curved-jordan-15657 was this the first time you turned on analytics? This usually happens when the index was created before the index template is created.
If the write operation to datahub_usage_event_v1 kicks in, es automatically creates the index with a guess at what each field should be.
We need the index template in before the index is created.
Can you 1) see if elasticsearch setup job ran without issues 2) try to access the es cluster and delete this index and restart gms
c
@early-lamp-41924 no i’ve used analytics before but maybe i did something wrong while upgrading datahub to the latest version. Also i had some problems and i had to delete indexes from es, delete datas from mysql and restart pods.
e
interesting. can you share the issues you had? That may have been the issue
c
I’ve checked the es setup job and saw datahub_usage_event_policy exists datahub_usage_event_index_template exists
I guess this tells something about it
@early-lamp-41924 i couldn’t see datahub_usage_event_v1 but i saw datahub_usage_event and datahub_usagestats_v1 in kibana. What should i do?
e
ah yes datahub_usage_event
you should delete
are you using aws es by any chance?
c
yes i use aws es
i just deleted the datahub_usage_event from es but don’t know how to restart gms, can you guide me somehow?
e
you shouldn’t have to
c
Btw, i’ve deleted that index but after i click the “analytics” button from right corner, es automatically creates the same index and it gives the same error
e
can you go to the website
any actions on it will generate a usage event
c
ok that makes sense, what should i do know?
e
but make sure
the index is deleted again
c
yeah i deleted the index and checked it
maybe i should delete datahub_usagestats_v1 index?
e
nope
that one is something different
c
Ok deleted the datahub_datahub_usage_event index again and i’m trying to access analytics page from website
Nope, same still
e
seems like the index template is not set up correctly
seems like you are using index prefix as datahub
is this correct?
c
yes thats true
also, es setup job includes these lines: - name: INDEX_PREFIX      value: datahub     - name: DATAHUB_ANALYTICS_ENABLED      value: “true”     - name: USE_AWS_ELASTICSEARCH      value: “true”
i think these are also correct
e
Yes this looks correct
do you have access to the es cluster?
can you try
Copy code
curl https://<<es-endpoint>>:<<port>>/_index_template/datahub_datahub_usage_event_index_template
c
I have access to the es cluster but couldn’t connect with curl. I guess it has authorization issue, how do i modify the curl command to use username and passwod?
e
that would be
username:pw@host:port !
c
{“error”{“root cause”[{“type”“resource not found exception”,“reason”“index template matching [datahub_datahub_usage_event_index_template] not found”}],“type”“resource not found exception”,“reason”“index template matching [datahub_datahub_usage_event_index_template] not found”},“status”:404}
e
this must be why
can you share the env variables used for elasticsearch setup job? shown on kubectl describe pod <<setup-job-pod-name>>
c
I hope i wrote the curl command properly
sure
e
oh if you’re not sure
you can try getting an index
curl <<host>>:<<port>>/datahub_datasetindex_v2
c
ok lets check
Environment:    ELASTICSEARCH_HOST:  <es-host>    ELASTICSEARCH_PORT:     443    ELASTICSEARCH_USE_SSL:   true    ELASTICSEARCH_USERNAME:   <username>    ELASTICSEARCH_PASSWORD:   <set to the key ‘password’ in secret ‘datahub-elasticsearch-secrets’> Optional: false    INDEX_PREFIX:        datahub    DATAHUB_ANALYTICS_ENABLED: true    USE_AWS_ELASTICSEARCH:   true
these are the envs
e
ahhh
you are using aws es
c
yepp
e
ahh i see the reason then
let’s try this
1. curl -XDELETE <<url>>/_opendistro/_ism/policies/datahub_datahub_usage_event_policy
2. curl -XDELETE <<url>>/_template/datahub_datahub_usage_event_index_template
and then rerun the setup job
c
Ok i’ve deleted these two
now i need to find out how to trigger the setup job from kubectl 😄 i’m kinda new at it
@early-lamp-41924 After loads of tries i resolved the issue with combining your helps. I need to write these steps: 1- Delete the datahub_usage_event index from es cluster 2- Run “curl -XDELETE <<url>>/_opendistro/_ism/policies/datahub_datahub_usage_event_policy” 3- Run “curl -XDELETE <<url>>/_template/datahub_datahub_usage_event_index_template” 4- Rerun the elasticsearch-setup-job 5- Check your setup job logs, if you have no errors, thats it! NOTE: Do not interact with website before all steps are done!!
e
awesome!!!!!
glad to hear
the problem was that
in the setup job, it creates a datahub_usage_event_00000 index to start off the process
but it doesn’t create if the template already exists
so when you deleted all the indices, this one got deleted so opendistro es didn’t have a place to start from
14 Views
Open in Slack
PreviousNext
Powered by