Hi - When deploying a fresh instance of Datahub v0...
# troubleshoot
e
Hi - When deploying a fresh instance of Datahub v0.10.0 via the helm charts(not upgrading), do we need to set
datahubUpgrade.enabled: false
? It defaults to true and we are running into an issue with the auth-secrets not existing yet and the job never running due to the pre-install helm hooks. Any ideas?
i
Hello Austin, Did you follow this guide? https://datahubproject.io/docs/deploy/kubernetes
e
Yes - we’ve been deploying without issue with v.0.8.0 and v0.9.0 - this is the first time we’ve had issues v0.10.0 and only with new instances of Datahub - there were no issues upgrading existing Datahub instances
Only thing I can see with the job is that the system-update-job has moved from post-install to pre-install with the helm hooks
o
DataHub Upgrade is a required component for GMS to run starting in 0.10.0. It ensures that the index mappings are compatible with the GMS deployment. If you disable it then GMS will never start up unless you configure it to bypass this check and then will run into errors with queries that expect mappings to be set up in certain ways.
If your auth secrets do not exist, you may need to change the deployment configuration for your set-up so that they are available prior to it executing.
a
I'm also running into similar issues with new AWS EKS deployment (and have followed the online deployment guides). I have to deploy the prerequisites and datahub without any custom values, then run the helm upgrade install with my custom values (for managed services) in order to deploy successfully. After all of that I'm getting a 500 error in the UI home page when datahubUpgrade.enabled = true I'm also running into elasticsearch-setup-job errors when datahub_analytics_enabled = true
o
What are you seeing in the GMS logs when you get the 500 in the UI? GMS hanging with no new logs? This would indicate a version mismatch between GMS & DataHub Upgrade or an unsuccessful run of Upgrade.
a
Copy code
2023-02-22 22:13:00,175 [ForkJoinPool.commonPool-worker-9] ERROR c.l.m.s.e.query.ESSearchDAO:61 - Search query failed
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=x_content_parse_exception, reason=[1:847] [bool] failed to parse field [must]]

        Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [HOST-REDACTED], URI [/corpuserindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parsing_exception","reason":"[match_phrase_prefix] query does not support [zero_terms_query]","line":1,"col":847}],"type":"x_content_parse_exception","reason":"[1:847] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:847] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:847] [bool] failed to parse field [should]","caused_by":{"type":"parsing_exception","reason":"[match_phrase_prefix] query does not support [zero_terms_query]","line":1,"col":847}}}},"status":400}

Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=x_content_parse_exception, reason=[1:847] [bool] failed to parse field [should]]

Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=parsing_exception, reason=[match_phrase_prefix] query does not support [zero_terms_query]]

2023-02-22 22:13:00,175 [ForkJoinPool.commonPool-worker-9] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to list users

Caused by: java.lang.RuntimeException: Failed to list users

Caused by: com.datahub.util.exception.ESQueryException: Search query failed:

Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=x_content_parse_exception, reason=[1:847] [bool] failed to parse field [must]]

2023-02-22 22:13:00,176 [ForkJoinPool.commonPool-worker-3] ERROR c.datahub.graphql.GraphQLController:101 - Errors while executing graphQL query: "query listUsers($input: ListUsersInput!) {\n  listUsers(input: $input) {\n    start\n    count\n    total\n    users {\n      urn\n      username\n      isNativeUser\n      info {\n        active\n        displayName\n        title\n        firstName\n        lastName\n        fullName\n        email\n        __typename\n      }\n      editableProperties {\n        displayName\n        pictureLink\n        teams\n        title\n        skills\n        __typename\n      }\n      status\n      roles: relationships(\n        input: {types: [\"IsMemberOfRole\"], direction: OUTGOING, start: 0}\n      ) {\n        start\n        count\n        total\n        relationships {\n          entity {\n            ... on DataHubRole {\n              urn\n              type\n              name\n              relationships(input: {types: [\"IsMemberOfRole\"], direction: INCOMING}) {\n                start\n                count\n                total\n                __typename\n              }\n              __typename\n            }\n            __typename\n          }\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[listUsers], extensions={code=500, type=SERVER_ERROR, classification=DataFetchingException}}], data={listUsers=null}, extensions={tracing={version=1, startTime=2023-02-22T22:13:00.141055Z, endTime=2023-02-22T22:13:00.176651Z, duration=35607463, parsing={startOffset=668358, duration=634575}, validation={startOffset=1197979, duration=501075}, execution={resolvers=[{path=[listUsers], parentType=Query, returnType=ListUsersResult, fieldName=listUsers, startOffset=1289604, duration=33412724}]}}}}, errors: [DataHubGraphQLError{path=[listUsers], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
Here are some snippets of the errors I'm seeing in the GMS logs when I go to the Users & Groups section. Looks like an error loading users and also the 500 error I mentioned before.
r
I was having a similar issue until I upgraded elasticsearch to > 7.10
a
Thanks for the suggestion, I'll give that a try!
Upgrading elasticsearch to 7.10 seems to have resolved the couple issues above, but the initial issue that Austin posted about still remains. I'm having to deploy how I mentioned in my first comment and I'm also having trouble with UI ingestion even once everything is installed. The request hits gms, but nothing seems to happen in the actions pod.
o
For the UI ingestion, the task shows that it is in running status in the UI and doesn't give any logs about the execution?
a
It's stuck in pending and nothing ever happens. Never gets to the running status.
b
hey James! so when you hit run, are you seeing a
dataHubExecutionRequest
entity being created in your document store? if that's being created, actions should be picking up on it as a new MCL.
a
Hey Chris! What service are you referring to when you say document store?
b
oh sorry! i meant in your mysql database
a
Ok, thanks for clarifying, I'll check
Yeah, I see all of the execution requests there, but they don't seem to have triggered the actions for some reason. I've also tried to play around with the actions versions some. I think default is 'v0.0.11' right now, but I currently have it set to 'v0.0.10' because I thought it might help to match it to the overall datahub release version of 'v0.10.0'. That being said the execution requests are all there, but have not worked for any version combination.
o
Actions follows a different release cadence from other DataHub components so v0.0.11 should be the correct version for DataHub release v0.10.0: https://github.com/acryldata/datahub-helm/blob/d68bf2c87c263e0d97446432b812df5cc561d21e/charts/datahub/values.yaml#L41
a
Thanks for the clarification. I was still having issues with actions v0.0.11 as well though
Anything else you guys can think of that I can check that would be preventing the actions pod from being triggered?