https://datahubproject.io logo
Join Slack
Powered by
Hello everyone - I've deployed the previous versio...
# getting-started
b
Hello everyone - I've deployed the previous version of datahub in AWS EKS Cluster - and all works good. However, when I deployed the new version (beta version) of datahub in AWS EKS Cluster - the front-end has the issues communicating with GMS server as an example as follows: Received error 404 from server for URI http://datahub-datahub-gms:8080/charts
i
Hello Peshan, What beta version are you talking about? Are you deploying via helm? Can you share your values.yaml?
b
Thank you so much, Padro for the quick response. Yes, it is via helm. The new version (pulling new images ) - and the analytics dashboard shows as "Analytics [beta]" And following this guide (last time also followed the same guide) for the values: https://datahubproject.io/docs/deploy/aws Thank you so much
i
What version of the images are you defining in the values.yaml?
0.8.42, latest?
b
Yes the prerequisites getting the values from here (default) - https://github.com/acryldata/datahub-helm/blob/master/charts/prerequisites/values.yaml And datahub is getting the values from here (defaullt) - https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml
i
Can you share the logs?
And run 
kubectl -n <namespace where datahub was deployed to> describe <gms pod id>
b
Sure - there is not errors in the log of gms pod However, there is errors in the front end pod: Caused by: com.linkedin.r2.message.rest.RestException: Received error 404 from server for URI http://datahub-datahub-gms:8080/corpUsers at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:76) ... 4 common frames omitted Caused by: com.linkedin.r2.message.rest.RestException: Received error 404 from server for URI http://datahub-datahub-gms:8080/datasets at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:76) ... 4 common frames omitted Caused by: com.linkedin.r2.message.rest.RestException: Received error 404 from server for URI http://datahub-datahub-gms:8080/dataJobs at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:76) ... 4 common frames omitted Caused by: com.linkedin.r2.message.rest.RestException: Received error 404 from server for URI http://datahub-datahub-gms:8080/dashboards at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:76) ... 4 common frames omitted Caused by: com.linkedin.r2.message.rest.RestException: Received error 404 from server for URI http://datahub-datahub-gms:8080/charts at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:76) ... 4 common frames omitted
This started happening only with the latest version (a couple of weeks ago, I deployed datahub and everything works without any issues) Looks like some integration issue between GUI and GMS Errors happening only within GUI pod and no errors showing showing in GMS
i
Please send a file with the output of 
kubectl describe <gms-pod>
and another file for 
kubectl logs gms-pod
What you sent is neither
Run the same command but for the frontend pod please
Please add gms and frontend logs, so far I see nothing wrong
b
Sure, please find logs for gms and frontend
@incalculable-ocean-74010 hope all are good with you. Were you able to see the errors in GUI logs? GUI won't able to render as it is not able to connect to GMS. How can we fix this? FYI, This is happening only with this new release and this works in the previous release.
m
@broad-river-92681: you seem to be running:
Copy code
Image:          linkedin/datahub-gms:v0.8.41
with
Copy code
Image:          linkedin/datahub-frontend-react:latest
is that by design? Ideally you should run the same tag for both. Typically in production you will want to use a release version, not 
latest
. Could you repeat this with the 
v0.8.41
tag for the frontend as well and let us know how it goes?
b
@mammoth-bear-12532 Thank you for the reply. Sure, we can change the front-end to v0.8.41 and update here the result. This is not by the design - this is how the default deployment scripts and instruction are pulling the tags of the images according to https://datahubproject.io/docs/deploy/aws/ Again, thank you so much.
@mammoth-bear-12532 This resolves the instability issues in the UI - now, UI is rendering good. However, I see some errors in UI also - then look at the frontend pod and seen these issues:
105648 [application-akka.actor.default-dispatcher-4] INFO o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0 105648 [application-akka.actor.default-dispatcher-4] INFO o.a.kafka.common.utils.AppInfoParser - Kafka commitId: fc1aaa116b661c8a 105648 [application-akka.actor.default-dispatcher-4] INFO o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1660561008937 105649 [kafka-producer-network-thread | datahub-frontend] INFO org.apache.kafka.clients.Metadata - [Producer clientId=datahub-frontend] Cluster ID: fFv15QHOQTW552UUjr3Gew 105718 [application-akka.actor.default-dispatcher-15] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-18] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-8] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-12] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-11] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-9] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-12] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-12] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105721 [application-akka.actor.default-dispatcher-9] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105745 [application-akka.actor.default-dispatcher-18] WARN p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check 105745 [application-akka.actor.default-dispatcher-18] INFO org.eclipse.jetty.util.log - Logging initialized @120244ms to org.eclipse.jetty.util.log.Slf4jLog 105745 [application-akka.actor.default-dispatcher-18] WARN o.e.j.j.spi.PropertyFileLoginModule - Exception starting propertyUserStore /etc/datahub/plugins/frontend/auth/user.props 105745 [application-akka.actor.default-dispatcher-18] ERROR application - The submitted callback is of type: class javax.security.auth.callback.NameCallback : javax.security.auth.callback.NameCallback@23fea581 105745 [application-akka.actor.default-dispatcher-18] ERROR application - The submitted callback is of type: class org.eclipse.jetty.jaas.callback.ObjectCallback : org.eclipse.jetty.jaas.callback.ObjectCallback@6df5aa57 105745 [application-akka.actor.default-dispatcher-18] WARN application - The submitted callback is unsupported! 105745 [application-akka.actor.default-dispatcher-18] ERROR application - The submitted callback is of type: class javax.security.auth.callback.PasswordCallback : javax.security.auth.callback.PasswordCallback@33e5ae32 105745 [application-akka.actor.default-dispatcher-18] ERROR application - The submitted callback is of type: class javax.security.auth.callback.NameCallback : javax.security.auth.callback.NameCallback@432b2ac 105745 [application-akka.actor.default-dispatcher-18] ERROR application - The submitted callback is of type: class org.eclipse.jetty.jaas.callback.ObjectCallback : org.eclipse.jetty.jaas.callback.ObjectCallback@29282a52 105745 [application-akka.actor.default-dispatcher-18] WARN application - The submitted callback is unsupported! 105745 [application-akka.actor.default-dispatcher-18] ERROR application - The submitted callback is of type: class javax.security.auth.callback.PasswordCallback : javax.security.auth.callback.PasswordCallback@3ba7dd5c
Thank you so much for the help and excellent support here
@mammoth-bear-12532 @incalculable-ocean-74010 This is one of the errors showing in the UI: Validation error of type FieldUndefined: Field 'statsSummary' in type 'Dataset' is undefined @ 'searchResultFields/statsSummary' (code undefined) Is this caused by mismatch of another pod image version? we can fix it and let us know, thank you so much
@mammoth-bear-12532 @incalculable-ocean-74010 FYI, these errors happening in the ingestion page Validation error of type FieldUndefined: Field 'id' in type 'ExecutionRequest' is undefined @ 'listIngestionSources/ingestionSources/executions/executionRequests/id' (code undefined)
Failed to load ingestion sources! Validation error of type FieldUndefined: Field 'id' in type 'ExecutionRequest' is undefined @ 'listIngestionSources/ingestionSources/executions/executionRequests/id'
@mammoth-bear-12532 @incalculable-ocean-74010 Would you be able to look into above and greatly appreciate for any resolution, many thanks. FYI, following are data helm values regarding the versions of images: Datahub https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml Prerequisites https://github.com/acryldata/datahub-helm/blob/master/charts/prerequisites/values.yaml
@mammoth-bear-12532 @incalculable-ocean-74010 Or shall I change the image tags to be latest for all the images and give a try? let me know which one would be the easiest way for us to resolve this, thank you so much.
m
@broad-river-92681 try moving to 0.8.43 and see if the issue remains.
The validation errors do indicate some sort of version mismatch, so try to make sure the pods are really running the version you are requesting
b
@mammoth-bear-12532 Thank you so much. Errors went away - it is working now.
m
glad to hear
3 Views
Open in Slack
PreviousNext
Powered by