https://datahubproject.io logo
Join Slack
Powered by
Hi folks, We've been trying to enable the analytic...
# troubleshoot
b
Hi folks, We've been trying to enable the analytics support in the DataHub UI and seem to be getting stuck hitting 500 due to DataFetchingExceptions. GMS logs indicate the index is not found:
Copy code
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
Details in 🧵
GMS stack:
Copy code
03:24:14 [ForkJoinPool.commonPool-worker-7] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler - Failed to execute DataFetcher
java.lang.RuntimeException: Search query failed:
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:265)
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getHighlights(AnalyticsService.java:236)
	at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.getHighlights(GetHighlightsResolver.java:50)
	at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:29)
	at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:22)
	at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:270)
	at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:203)
	at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:60)
	at graphql.execution.Execution.executeOperation(Execution.java:165)
	at graphql.execution.Execution.execute(Execution.java:104)
	at graphql.GraphQL.execute(GraphQL.java:557)
	at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:482)
	at graphql.GraphQL.executeAsync(GraphQL.java:446)
	at graphql.GraphQL.execute(GraphQL.java:377)
	at com.linkedin.datahub.graphql.GraphQLEngine.execute(GraphQLEngine.java:88)
	at com.datahub.graphql.GraphQLController.lambda$postGraphQL$0(GraphQLController.java:89)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
	at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:260)
	... 22 common frames omitted
We have run the ES setup scripts, I have checked that the templates exist on our ES setup:
Copy code
$ curl  "<http://localhost:9200/_ilm/policy/datahub_usage_event_policy>" | jq
{
  "datahub_usage_event_policy": {
    "version": 1,
    "modified_date": "2022-08-14T20:35:15.719Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "7d"
            }
          }
        },
        "delete": {
          "min_age": "60d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      }
    }
  }
}
and
Copy code
$ curl "<http://localhost:9200/_index_template/datahub_usage_event_index_template>" | jq
{
  "index_templates": [
    {
      "name": "datahub_usage_event_index_template",
      "index_template": {
        "index_patterns": [
          "*datahub_usage_event*"
        ],
        "template": {
          "settings": {
            "index": {
              "lifecycle": {
                "name": "datahub_usage_event_policy"
              }
            }
          },
          "mappings": {
            "properties": {
              "browserId": {
                "type": "keyword"
              },
              "@timestamp": {
                "type": "date"
              },
              "userAgent": {
                "type": "keyword"
              },
              "type": {
                "type": "keyword"
              },
              "timestamp": {
                "type": "date"
              }
            }
          }
        },
        "composed_of": [],
        "priority": 500,
        "data_stream": {}
      }
    }
  ]
}
We do see our MAE consumer running and I added some logging and I do see the usage events being processed:
Copy code
02:54:22 [datahub-usage-event-consumer-job-client-0-C-1] INFO  c.l.m.k.DataHubUsageEventsProcessor - Got usage event: {"type":"SearchResultsViewEvent","query":"ice","total":10,"actorUrn":"urn:li:corpuser:datahub","timestamp":1660618462854,"date":"Mon Aug 15 2022 19:54:22 GMT-0700 (Pacific Daylight Time)","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36","browserId":"f561d935-96a2-410c-a8e0-1baab1b83787"}
Looking on the ES side, I don't seem to find any indices matching usage_events:
Copy code
$ curl 'localhost:9200/_cat/indices' | grep usage
green open dataset_datasetusagestatisticsaspect_v1_1658472208943                  Ugl-dJDnQum0hp_Q6WZ_rA 1 1       0      0    416b    208b
green open dashboard_dashboardusagestatisticsaspect_v1                            WOVOEFcVR56OsTt4oTA6wQ 1 1       0      0    416b    208b
So it seems like for some reason the 
DataHubUsageEventsProcessor
is not creating these indices.
We have tried restarting the MAE consumer (and GMS as well) but we don't seem to be able to trigger the creation of the 
datahub_usage_event
index.
Could someone clarify if they are meant to be created at event processing time (this is my understanding) or at MAE consumer startup time? And any suggestions on how we can debug further?
b
Hi Piyush , looks like indices were not properly created . You can follow below thread where user faced same issue https://datahubspace.slack.com/archives/C029A3M079U/p1646431113049059
High level steps would be - 1. Stop datahub-gms 2. Delete the datahub_usage_event index from es cluster Run “curl -XDELETE <<url>>/_opendistro/_ism/policies/datahub_datahub_usage_event_policy” 3. Delete the datahub_usage_event template from es cluster Run “curl -XDELETE <<url>>/_template/datahub_datahub_usage_event_index_template” 4. Rerun the elasticsearch-setup-job 5. Check your setup job logs, if you have no errors, thats it! 6. Restart datahub-gms NOTE: Do not interact with website before all steps are done!!
b
Hi @bumpy-needle-3184 yeah I've read through a few of those threads. What I'm seeing is the indices aren't being created at all. So I'm trying to figure out why the indices aren't being created at all when we have the templates and the policy successfully written out and the MAE consumer job running. https://datahubspace.slack.com/archives/C029A3M079U/p1660620729075069?thread_ts=1660620440.311029&amp;cid=C029A3M079U
Circling back, I was able hit the simulate endpoint and see that the indices were setup correctly. So I just tried to create them via curl to see what happens:
Copy code
curl -X PUT "<http://sproxy.service.envoy:10080/searchdatahub/_data_stream/datahub_usage_event?pretty>"
{
  "acknowledged" : true
}
and then I see the index created:
Copy code
curl 'localhost:9200/_cat/indices'
...
green open .ds-datahub_usage_event-000001                                         2QPTOwFeQbCV_qI9fdEOQw 1 1      33      0 258.2kb 145.6kb
...
It looks like the app's analytics tab is functional now 🤷‍♂️ Not sure if the create-indices script should include a step to explicitly create the 
datahub_usage_event
index aside from the template and ilm policy..
b
index are supposd to be created by gms instance adding @square-activity-64562 for more insight
b
Yeah would love to learn more on this. I hunted around and wasn't able to track how the datahub_usage_event index is being created. A bunch of others seem to be created at started in the EsIndexBuilders based on entities / aspects in the entity registry. I think the usage_event index is created seemingly based on inserts as part of the DataHubUsageEventsProcessor job. But I wasn't able to figure out why this wasn't happening in my case event though the index templates existed and we had events being processed by this job and hence being written out to ES. After I manually created the index with the curl, I am seeing the analytics data so I know the job etc is working fine. Just the initial index creation which held this up
5 Views
Open in Slack
PreviousNext
Powered by