https://datahubproject.io logo
Join Slack
Powered by
How do I set/override the name/group id of "generi...
# troubleshoot
c
How do I set/override the name/group id of "generic-mce-consumer-job-client" and "generic-mae-consumer-job-client" when using the DataHub Helm Charts? I'm trying to enable the analytics feature but I'm getting the following error as Kafka group ids have to be prefixed in my organization: 
07:51:34.891 [generic-mce-consumer-job-client-0-C-1[] ERROR o.s.k.l.KafkaMessageListenerContainer$ListenerConsumer - Authorization Exception
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: generic-mce-consumer-job-client
Which I think in turn is causing: 
07:51:34.903 [generic-mce-consumer-job-client-0-C-1[] ERROR o.s.k.l.KafkaMessageListenerContainer$ListenerConsumer - Fatal consumer exception; stopping container
And perhaps finally causing the following error when trying to access the analytics tab:
Copy code
08:17:15.790 [Thread-13[] ERROR c.l.d.g.a.service.AnalyticsService - Search query failed: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
08:17:15.792 [Thread-13[] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler - Failed to execute DataFetcher
java.lang.RuntimeException: Search query failed:
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:245)
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getTimeseriesChart(AnalyticsService.java:93)
    at com.linkedin.datahub.graphql.analytics.resolver.GetChartsResolver.getProductAnalyticsCharts(GetChartsResolver.java:59)
    at com.linkedin.datahub.graphql.analytics.resolver.GetChartsResolver.get(GetChartsResolver.java:39)
    at com.linkedin.datahub.graphql.analytics.resolver.GetChartsResolver.get(GetChartsResolver.java:27)
    at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:270)
    at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:203)
    at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:60)
    at graphql.execution.Execution.executeOperation(Execution.java:165)
    at graphql.execution.Execution.execute(Execution.java:104)
    at graphql.GraphQL.execute(GraphQL.java:557)
    at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:482)
    at graphql.GraphQL.executeAsync(GraphQL.java:446)
    at graphql.GraphQL.execute(GraphQL.java:377)
    at com.linkedin.datahub.graphql.GraphQLEngine.execute(GraphQLEngine.java:88)
    at com.datahub.metadata.graphql.GraphQLController.lambda$postGraphQL$0(GraphQLController.java:82)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:240)
    ... 17 common frames omitted
    Suppressed: org.elasticsearch.client.ResponseException: method [POST[], host [<https://search-datahub-cjfy6-y4e4cliwdivgdm3iou375f7d4a.eu-central-1.es.amazonaws.com:443>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"}],"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"},"status":404}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 21 common frames omitted
08:17:15.798 [Thread-15[] ERROR c.l.d.g.a.service.AnalyticsService - Search query failed: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
08:17:15.802 [Thread-13[] ERROR c.d.m.graphql.GraphQLController - Errors while executing graphQL query: "query getAnalyticsCharts {\n  getAnalyticsCharts {\n    title\n    charts {\n      ... on TimeSeriesChart {\n        title\n        lines {\n          name\n          data {\n            x\n            y\n            __typename\n          }\n          __typename\n        }\n        dateRange {\n          start\n          end\n          __typename\n        }\n        interval\n        __typename\n      }\n      ... on BarChart {\n        title\n        bars {\n          name\n          segments {\n            label\n            value\n            __typename\n          }\n          __typename\n        }\n        __typename\n      }\n      ... on TableChart {\n        title\n        columns\n        rows {\n          values\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[getAnalyticsCharts[], extensions={code=500, classification=DataFetchingException}}], data=null}, errors: [DataHubGraphQLError{path=[getAnalyticsCharts[], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
08:17:15.799 [Thread-15[] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler - Failed to execute DataFetcher
java.lang.RuntimeException: Search query failed:
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:245)
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getHighlights(AnalyticsService.java:216)
    at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.getHighlights(GetHighlightsResolver.java:50)
    at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:29)
    at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:19)
    at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:270)
    at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:203)
    at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:60)
    at graphql.execution.Execution.executeOperation(Execution.java:165)
    at graphql.execution.Execution.execute(Execution.java:104)
    at graphql.GraphQL.execute(GraphQL.java:557)
    at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:482)
    at graphql.GraphQL.executeAsync(GraphQL.java:446)
    at graphql.GraphQL.execute(GraphQL.java:377)
    at com.linkedin.datahub.graphql.GraphQLEngine.execute(GraphQLEngine.java:88)
    at com.datahub.metadata.graphql.GraphQLController.lambda$postGraphQL$0(GraphQLController.java:82)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:240)
    ... 17 common frames omitted
    Suppressed: org.elasticsearch.client.ResponseException: method [POST[], host [<https://search-datahub-cjfy6-y4e4cliwdivgdm3iou375f7d4a.eu-central-1.es.amazonaws.com:443>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"}],"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"},"status":404}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 21 common frames omitted
08:17:15.807 [Thread-15[] ERROR c.d.m.graphql.GraphQLController - Errors while executing graphQL query: "query getHighlights {\n  getHighlights {\n    value\n    title\n    body\n    __typename\n  }\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[getHighlights[], extensions={code=500, classification=DataFetchingException}}], data=null}, errors: [DataHubGraphQLError{path=[getHighlights[], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
I've tried setting the following environment variables: 
DATAHUB_USAGE_EVENT_NAME
DATAHUB_USAGE_EVENT_KAFKA_CONSUMER_GROUP_ID
DATAHUB_TRACKING_TOPIC
⬆️ 1
e
For the elasticsearch issue, can you check the logs of the elasticsearchSetupJob? Also, are you using AWS managed elasticsearch by any chance?
Not sure if setting group-id would help but these are the two env variables to control consumer group ids 
METADATA_CHANGE_LOG_KAFKA_CONSUMER_GROUP_ID
and 
METADATA_CHANGE_PROPOSAL_KAFKA_CONSUMER_GROUP_ID
c
Thanks Dexter. Yes I'll try the env vars first and then check the elasticsearch setup job logs if that doesn't do the trick. And yes I'm using AWS managed ElasticSearch.
Hi @early-lamp-41924, the env vars fixed the unauthed errors in the GMS pod and the ElasticSearch Setup Job log look good but the analytics tab still returns 500 and produces the following error in the GMS log:
Copy code
12:12:56.455 [Thread-18[] ERROR c.l.d.g.a.service.AnalyticsService - Search query failed: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
12:12:56.455 [Thread-18[] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler - Failed to execute DataFetcher
java.lang.RuntimeException: Search query failed:
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:245)
...    at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
...    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:240)
    ... 17 common frames omitted
    Suppressed: org.elasticsearch.client.ResponseException: method [POST[], host [<https://search-datahub-cjfy6-y4e4cliwdivgdm3iou375f7d4a.eu-central-1.es.amazonaws.com:443>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"}],"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"},"status":404}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 21 common frames omitted
12:12:56.456 [Thread-18[] ERROR c.d.m.graphql.GraphQLController - Errors while executing graphQL query: "query getAnalyticsCharts {\n  getAnalyticsCharts {\n    title\n    charts {\n      ... on TimeSeriesChart {\n        title\n        lines {\n          name\n          data {\n            x\n            y\n            __typename\n          }\n          __typename\n        }\n        dateRange {\n          start\n          end\n          __typename\n        }\n        interval\n        __typename\n      }\n      ... on BarChart {\n        title\n        bars {\n          name\n          segments {\n            label\n            value\n            __typename\n          }\n          __typename\n        }\n        __typename\n      }\n      ... on TableChart {\n        title\n        columns\n        rows {\n          values\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[getAnalyticsCharts[], extensions={code=500, classification=DataFetchingException}}], data=null}, errors: [DataHubGraphQLError{path=[getAnalyticsCharts[], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
12:12:56.509 [Thread-20[] ERROR c.l.d.g.a.service.AnalyticsService - Search query failed: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
12:12:56.509 [Thread-20[] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler - Failed to execute DataFetcher
java.lang.RuntimeException: Search query failed:
    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:245)
...    at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event[]]
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
...    at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:240)
    ... 17 common frames omitted
    Suppressed: org.elasticsearch.client.ResponseException: method [POST[], host [<https://search-datahub-cjfy6-y4e4cliwdivgdm3iou375f7d4a.eu-central-1.es.amazonaws.com:443>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"}],"type":"index_not_found_exception","reason":"no such index [datahub_usage_event[]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"},"status":404}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 21 common frames omitted
12:12:56.511 [Thread-20[] ERROR c.d.m.graphql.GraphQLController - Errors while executing graphQL query: "query getHighlights {\n  getHighlights {\n    value\n    title\n    body\n    __typename\n  }\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[getHighlights[], extensions={code=500, classification=DataFetchingException}}], data=null}, errors: [DataHubGraphQLError{path=[getHighlights[], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
I think it is partly due to this frontend error:
Copy code
12:16:44 [application-akka.actor.default-dispatcher-289[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Type: application/json' is ignored, explicit `Content-Type` header is not allowed. Set `HttpResponse.entity.contentType` instead.
12:16:44 [application-akka.actor.default-dispatcher-289[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Length: 8014' is ignored, explicit `Content-Length` header is not allowed. Use the appropriate HttpEntity subtype.
12:16:44 [kafka-producer-network-thread | datahub-frontend] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=datahub-frontend] Error while fetching metadata with correlation id 15 : {DataHubUsageEvent_v1=TOPIC_AUTHORIZATION_FAILED}
12:16:44 [kafka-producer-network-thread | datahub-frontend] ERROR org.apache.kafka.clients.Metadata - [Producer clientId=datahub-frontend] Topic authorization failed for topics [DataHubUsageEvent_v1[]
12:16:44 [application-akka.actor.default-dispatcher-290[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Type: application/json' is ignored, explicit `Content-Type` header is not allowed. Set `HttpResponse.entity.contentType` instead.
12:16:44 [application-akka.actor.default-dispatcher-290[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Type: application/json' is ignored, explicit `Content-Type` header is not allowed. Set `HttpResponse.entity.contentType` instead.
12:16:44 [application-akka.actor.default-dispatcher-290[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Length: 198' is ignored, explicit `Content-Length` header is not allowed. Use the appropriate HttpEntity subtype.
12:16:44 [application-akka.actor.default-dispatcher-290[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Length: 193' is ignored, explicit `Content-Length` header is not allowed. Use the appropriate HttpEntity subtype.
12:16:44 [application-akka.actor.default-dispatcher-283[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Type: application/json' is ignored, explicit `Content-Type` header is not allowed. Set `HttpResponse.entity.contentType` instead.
12:16:44 [application-akka.actor.default-dispatcher-283[] WARN  akka.actor.ActorSystemImpl - Explicitly set HTTP header 'Content-Length: 612' is ignored, explicit `Content-Length` header is not allowed. Use the appropriate HttpEntity subtype.
Do you also know the env var for setting the DataHubUsageEvent topic name in the frontend? Here is the ElasticSearchSetupJob logs:
Copy code
2021/10/19 12:09:51 Waiting for: <https://user:password@search-datahub-cjfy6-y4e4cliwdivgdm3iou375f7d4a.eu-central-1.es.amazonaws.com:443>
2021/10/19 12:09:52 Received 200 from <https://user:password@search-datahub-cjfy6-y4e4cliwdivgdm3iou375f7d4a.eu-central-1.es.amazonaws.com:443>
datahub_usage_event_policy exists
creating datahub_usage_event_index_template
{
  "index_patterns": ["*datahub_usage_event*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "type": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date"
        },
        "userAgent": {
          "type": "keyword"
        },
        "browserId": {
          "type": "keyword"
        }
      }
    },
    "settings": {
      "index.lifecycle.name": "datahub_usage_event_policy"
    }
  }
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1167  100   647  100   520   3778   3036 --:--:-- --:--:-- --:--:--  6824
2021/10/19 12:09:52 Command finished successfully.
}{"error":{"root_cause":[{"type":"invalid_index_template_exception","reason":"index_template [datahub_usage_event_index_template] invalid, cause [Validation Failed: 1: unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings;]"}],"type":"invalid_index_template_exception","reason":"index_template [datahub_usage_event_index_template] invalid, cause [Validation Failed: 1: unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings;]"},"status":400}
e
Very interesting. So you set USE_AWS_ELASTICSEARCH to true for the elasticsearch setup job right?
c
No, I had wrongfully assumed https://datahubproject.io/docs/deploy/aws was for deploying DataHub and only DataHub to another AWS service than a K8s service and therefore always ignored it and only looked at https://datahubproject.io/docs/deploy/kubernetes. Realizing what the first guide actually contained could have saved me days of debugging. It works now. Thank you so much, @early-lamp-41924.
🙌 2
e
Glad it worked out!!!
7 Views
Open in Slack
PreviousNext
Powered by