On another topic I keep getting the following error in the G DataHub #troubleshoot

On another topic, I keep getting the following err...

green-hamburger-3800

11/14/2022, 3:19 PM

On another topic, I keep getting the following error in the GMS logs:

Copy code

15:18:28.491 [qtp1830908236-10] WARN  c.d.a.a.AuthenticatorChain:70 - Authentication chain failed to resolve a valid authentication. Errors: [(com.datahub.authentication.authenticator.DataHubSystemAuthenticator,Failed to authenticate inbound request: Authorization header is missing Authorization header.), (com.datahub.authentication.authenticator.DataHubTokenAuthenticator,Failed to authenticate inbound request: Request is missing 'Authorization' header.)]

and I'm not sure where is it coming from since no ingestion/usage exists at the moment (I'd guess an internal thing and I might be missing some configuration?) Thanks (=

plus1 1

echoing-airport-49548

11/14/2022, 6:43 PM

Let me look into this and get back to you

green-hamburger-3800

11/15/2022, 8:49 AM

Thanks a lot!

silly-angle-91497

02/10/2023, 9:57 PM

Was there ever a resolution to this problem? I'm experiencing the same issue when I upgraded from v0.8.45 to v0.10.0

echoing-airport-49548

02/10/2023, 10:13 PM

Hey @silly-angle-91497 where are you seeing this issue?

echoing-airport-49548

02/10/2023, 10:13 PM

Is it causing issues with using the app?

silly-angle-91497

02/10/2023, 10:15 PM

So i'm looking at the pods logs using kubectl. One second and i'll get you a screenshot. I changed the value for global.datahub.metadata_service_authentication.enabled to false to see if that would fix the issue. i'm rerunning the helm upgrad now.

silly-angle-91497

02/10/2023, 10:19 PM

Ok, after i disabled the authentication (which I really don't want to do), I don't see those errors anymore. But I'm still getting the Code 500 in the UI when searching for entities (nothing comes up). When trailing the GMS pod logs I am seeing this:

Copy code

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<https://search-datahub-cihpbucl53m35jsrl3nlj5a2vm.us-west-2.es.amazonaws.com:443>], URI [/datasetindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parsing_exception","reason":"[match_phrase_prefix] query does not support [zero_terms_query]","line":1,"col":1932}],"type":"x_content_parse_exception","reason":"[1:1932] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:1932] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:1932] [bool] failed to parse field [should]","caused_by":{"type":"parsing_exception","reason":"[match_phrase_prefix] query does not support [zero_terms_query]","line":1,"col":1932}}}},"status":400}
                at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
                at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
                ... 16 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=x_content_parse_exception, reason=[1:1932] [bool] failed to parse field [should]]
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
        at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
        at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603)
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
        ... 19 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=x_content_parse_exception, reason=[1:1932] [bool] failed to parse field [should]]
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
        at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
        ... 23 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=parsing_exception, reason=[match_phrase_prefix] query does not support [zero_terms_query]]
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
        at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
        ... 25 common frames omitted

silly-angle-91497

02/10/2023, 10:20 PM

Here are the running pods:

Copy code

 ~/git/gd-glass/ [main*] kubectl get pods                                   
NAME                                               READY   STATUS      RESTARTS   AGE
datahub-acryl-datahub-actions-694c45bb98-fm5xw     1/1     Running     0          6m44s
datahub-datahub-frontend-75cd5f77d7-9hp6x          1/1     Running     0          6m44s
datahub-datahub-gms-fd97f8946-v86zn                1/1     Running     0          6m43s
datahub-datahub-system-update-job-9rpxh            0/1     Completed   0          8m53s
datahub-elasticsearch-setup-job-7wg7v              0/1     Completed   0          12m
datahub-kafka-setup-job-cwz9w                      0/1     Completed   0          12m
datahub-mysql-setup-job-97nd2                      0/1     Completed   0          9m26s
datahub-nocode-migration-job-5j4jv                 0/1     Completed   0          6m42s
prerequisites-cp-schema-registry-cf79bfccf-kc2pm   2/2     Running     0          5h9m
prerequisites-kafka-0                              1/1     Running     0          5h9m
prerequisites-zookeeper-0                          1/1     Running     0          5h9m

silly-angle-91497

02/10/2023, 10:20 PM

And I was trailing this pod: datahub-datahub-gms-fd97f8946-v86zn

silly-angle-91497

02/10/2023, 10:22 PM

I'm going to re-enable the authentication now since I don't think that's causing the UI Code 500

green-hamburger-3800

02/13/2023, 4:06 PM

Never looked at it again since the app was usable. Decided to forget about it 😅 Just checked the logs and it's still happening though sadpanda (But no issues when using the app)

quiet-television-68466

03/09/2023, 11:13 AM

We’re having the same issue here! Interested in a resolution as we want to start ingesting our logs into external software for monitoring purposes.

millions-musician-14958

03/09/2023, 1:18 PM

Same issue here after upgrading to v0.10.0 😕 I am able to ingest data from various sources (Redshift, Glue), and I am also able to use Datahub UI, so I am not sure how much this error hurts my users. All I know is that there are hundreds of these messages in the logs.

millions-musician-14958

03/09/2023, 1:28 PM

@quiet-television-68466 could you try running these two commands:

Copy code

kubectl exec -n <<namespace>> --stdin --tty <<frontend-pod-name>> -- /bin/sh

curl --location --request POST '<http://datahub-datahub-gms:8080/entities?action=searchAcrossEntities>' \
--header 'X-RestLi-Protocol-Version: 2.0.0' \
--header 'Content-Type: application/json' \
--data-raw '{
    "entities": ["dataset"],
    "input": "*",
    "start": 0,
    "count": 1000
}'

millions-musician-14958

03/09/2023, 1:28 PM

I am getting a HTML page with the following error:

HTTP ERROR 401 Unauthorized to perform this action.

I suppose you will see it too.

quiet-television-68466

03/09/2023, 1:33 PM

Exact same error @millions-musician-14958

millions-musician-14958

03/09/2023, 1:36 PM

OK, I assume that that’s because we have

metadata_service_authentication

enabled in the configuration. There is some other service that’s hitting GMS with unauthorized requests. Now the question is which service is it.

green-hamburger-3800

03/09/2023, 1:37 PM

Indeed the error started when I enabled the

metadata_service_authentication

plus1 1

millions-musician-14958

03/09/2023, 1:38 PM

I can use DataHub UI just fine so I assume it’s not about the requests generated by the browser

green-hamburger-3800

03/09/2023, 1:41 PM

I didn't have any issues with the UI/Ingestion/CLI since on all cases either you authenticate and grab a session or pass a token. So I'd guess it'd be an inner service, but It seems to be working fine overall 🤷

millions-musician-14958

03/09/2023, 1:53 PM

Do you run Prometheus on your cluster?

millions-musician-14958

03/09/2023, 2:05 PM

In my case, I can see that Prometheus is hitting port 8080 (http-alt) of GMS, although Prometheus is not configured to access GMS (apparently):

millions-musician-14958

03/09/2023, 2:06 PM

This could explain some of these messages

green-hamburger-3800

03/09/2023, 2:38 PM

I don't have any Prometheus hitting it as far as I could see 🤔

hallowed-lizard-92381

04/12/2023, 4:48 PM

Having same issue after upgrading to 10.0.X Were any of you able to resolve or pin down the reason?

freezing-application-4446

08/23/2023, 9:46 AM

in my case, this started appearing after I set up an ingress to the GMS service via an AWS ALB. I resolved it by modifying the health check path for the target group:

/config

green-hamburger-3800

08/24/2023, 8:13 AM

Uh, that’s some good info! Thanks a lot! I’ll check that out (=

rough-spring-71950

10/18/2023, 4:30 PM

Did anyone get a resolution here?

73 Views

Open in Slack

Previous Next