Hello, I’m trying to upgrade DataHub from v0.8.21 to v0.8.23 (and eventually to the latest) and running into UI login error as default “datahub” user upon initial upgrade. I’ll include more details in the thread. Any help would be greatly appreciated. Thanks!

This @gray-carpet-60705 - It appears that the backend (datahub-gms) is down. Can you confirm whether that container is up and running??

Hi @big-carpet-38439, thanks for the reply. gms and mce/mae consumers all show up as running, but they don’t appear to be responsive. No log is available for mce, and gms log stopped after the following. I’m installing Helm chart version 0.2.41 with image tag v0.8.23, and we are on AWS using EKS, RDS Postgres. MSK, Glue schema registry, and ES. Is there something else I can try to get gms back up?

20:22:02.902 [main] INFO  c.a.s.s.c.c.GlueSchemaRegistryConfiguration:188 - Cache Time to live is not found, using default 86400000
20:22:04.107 [main] INFO  o.a.kafka.common.utils.AppInfoParser:117 - Kafka version: 5.5.1-ccs
20:22:04.108 [main] INFO  o.a.kafka.common.utils.AppInfoParser:118 - Kafka commitId: cb1873c1fdf5f5f9
20:22:04.108 [main] INFO  o.a.kafka.common.utils.AppInfoParser:119 - Kafka startTimeMs: 1647289324103
20:22:04.320 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - dataHubPolicyKey schema is compatible with previous schema due to
20:22:04.322 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - dataHubPolicyInfo schema is compatible with previous schema due to
20:22:04.322 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserKey schema is compatible with previous schema due to
20:22:04.323 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserEditableInfo schema is compatible with previous schema due to
20:22:04.323 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserInfo schema is compatible with previous schema due to
20:22:04.324 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserStatus schema is compatible with previous schema due to
20:22:04.324 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - dataJobKey schema is compatible with previous schema due to
20:22:04.325 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - datasetKey schema is compatible with previous schema due to
20:22:04.325 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - viewProperties schema is compatible with previous schema due to
20:22:04.328 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - schemaMetadata schema is compatible with previous schema due to
20:22:04.329 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - status schema is compatible with previous schema due to
20:22:04.978 [kafka-producer-network-thread | producer-1] INFO  org.apache.kafka.clients.Metadata:277 - [Producer clientId=producer-1] Cluster ID: ...
20:22:04.978 [kafka-producer-network-thread | producer-1] INFO  org.apache.kafka.clients.Metadata:277 - [Producer clientId=producer-1] Cluster ID: ...
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8Mar 14, 2022 8:22:05 PM org.neo4j.driver.internal.logging.JULogger info
INFO: Direct driver instance 76314483 created for server address localhost:7687
20:22:05.630 [main] WARN  c.l.r.t.h.client.HttpClientFactory:894 - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
20:22:05.632 [main] WARN  c.l.r.t.h.client.HttpClientFactory:903 - No callback executor is provided to HttpClientFactory, using it's own call back executor.
20:22:05.633 [main] WARN  c.l.r.t.h.client.HttpClientFactory:911 - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
20:22:05.647 [main] INFO  c.l.r.t.h.client.HttpClientFactory:1338 - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 '
20:22:05.969 [main] WARN  c.l.r.t.h.client.HttpClientFactory:894 - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
20:22:05.969 [main] WARN  c.l.r.t.h.client.HttpClientFactory:903 - No callback executor is provided to HttpClientFactory, using it's own call back executor.
20:22:05.969 [main] WARN  c.l.r.t.h.client.HttpClientFactory:911 - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
20:22:05.970 [main] INFO  c.l.r.t.h.client.HttpClientFactory:1338 - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 '
20:22:06.495 [main] INFO  c.l.r.s.c.ResourceMethodConfigProviderImpl:75 - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
20:22:06.517 [main] INFO  c.l.r.s.c.ResourceMethodConfigProviderImpl:75 - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
20:22:06.519 [main] INFO  c.l.r.s.c.ResourceMethodConfigProviderImpl:75 - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
20:22:07.835 [main] INFO  c.l.g.f.k.SimpleKafkaConsumerFactory:47 - Simple KafkaListenerContainerFactory built successfully
20:22:07.856 [main] INFO  c.l.g.f.k.KafkaEventConsumerFactory:82 - Event-based KafkaListenerContainerFactory built successfully
20:22:07.859 [main] INFO  c.l.g.f.k.s.KafkaSchemaRegistryFactory:58 - creating schema registry config using url: <http://prerequisites-cp-schema-registry:8081>
20:22:07.912 [main] INFO  c.l.metadata.boot.BootstrapManager:24 - Starting Bootstrap Process...
20:22:07.912 [main] INFO  c.l.metadata.boot.BootstrapManager:28 - Executing bootstrap step 1/5 with name IngestRootUserStep...
20:22:08.684 [main] INFO  c.l.metadata.boot.BootstrapManager:28 - Executing bootstrap step 2/5 with name IngestPoliciesStep...
20:22:08.685 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:49 - Ingesting default access policies...
20:22:08.698 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:0
20:22:08.789 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:1
20:22:08.842 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:2
20:22:08.881 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:3
20:22:08.924 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:4
20:22:08.975 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:5
20:22:09.008 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:6
20:22:09.048 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:77 - Skipping ingestion of editable policy with urn urn:li:dataHubPolicy:7
20:22:09.048 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:81 - Successfully ingested default access policies.
20:22:09.049 [main] INFO  c.l.metadata.boot.BootstrapManager:28 - Executing bootstrap step 3/5 with name IngestDataPlatformsStep...
20:22:09.807 [main] INFO  c.l.metadata.boot.BootstrapManager:36 - Starting asynchronous bootstrap step 4/5 with name IngestDataPlatformInstancesStep...
20:22:09.812 [main] INFO  c.l.metadata.boot.BootstrapManager:36 - Starting asynchronous bootstrap step 5/5 with name IngestRetentionPoliciesStep...
20:22:09.812 [Thread-13] INFO  c.l.m.b.s.IngestDataPlatformInstancesStep:61 - Checking for DataPlatformInstance
20:22:09.813 [Thread-14] INFO  c.l.m.b.s.IngestRetentionPoliciesStep:47 - Ingesting default retention...
20:22:09.813 [Thread-14] INFO  c.l.m.b.s.IngestRetentionPoliciesStep:51 - IngestRetentionPolicies disabled. Skipping.
20:22:09.818 [Thread-13] INFO  c.l.m.b.s.IngestDataPlatformInstancesStep:67 - DataPlaformInstance aspect exists. Skipping step
20:22:09.822 [main] INFO  o.s.web.context.ContextLoader:307 - Root WebApplicationContext initialized in 16887 ms
20:22:09.858 [main] INFO  c.d.a.filter.AuthenticationFilter:143 - Auth is disabled. Building no-op authenticator chain...
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.82022-03-14 20:22:10.002:INFO:oejshC.ROOT:main: Initializing Spring DispatcherServlet 'apiServlet'
20:22:10.002 [main] INFO  o.s.web.servlet.DispatcherServlet:525 - Initializing Servlet 'apiServlet'
20:22:10.070 [main] ERROR c.datahub.graphql.GraphQLController:32 - I created graphqlcontroller
20:22:10.362 [main] WARN  o.s.h.c.j.Jackson2ObjectMapperBuilder:127 - For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
20:22:10.598 [main] INFO  o.s.web.servlet.DispatcherServlet:547 - Completed initialization in 596 ms
2022-03-14 20:22:10.599:INFO:oejshC.ROOT:main: Initializing Spring DispatcherServlet 'authServlet'
20:22:10.599 [main] INFO  o.s.web.servlet.DispatcherServlet:525 - Initializing Servlet 'authServlet'
20:22:10.673 [main] INFO  o.s.web.servlet.DispatcherServlet:547 - Completed initialization in 74 ms
2022-03-14 20:22:10.674:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@60215eee{Open source GMS,/,file:///tmp/jetty-0_0_0_0-8080-war_war-_-any-5633708735902109725.dir/webapp/,AVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
2022-03-14 20:22:10.747:INFO:oejs.AbstractConnector:main: Started ServerConnector@6ed3ef1{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2022-03-14 20:22:10.747:INFO:oejs.Server:main: Started @29727ms

Oh interesting.. These logs seem to indicate that GMS is indeed up!

hmn seems like it started fine. wonder why frontend is unable to reach it. Can you try portforwarding

kubectl port-forward svc/datahub-datahub-gms 8080:8080

This is the Istio gateway and virtual service that we have originally set up. Not sure if these rewrites are causing any problem.

apiVersion: <http://networking.istio.io/v1alpha3|networking.istio.io/v1alpha3>
kind: Gateway
metadata:
  name: datahub-gateway
spec:
  selector:
    istio: datahub-ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: datahub-tls-secret
      hosts:
        - {{ .Values.dataHub.hostname | quote }}
---
apiVersion: <http://networking.istio.io/v1alpha3|networking.istio.io/v1alpha3>
kind: VirtualService
metadata:
  name: datahub-service
spec:
  hosts:
    - {{ .Values.dataHub.hostname | quote }}
  gateways:
    - datahub-gateway
  http:
    # Routes for gms ingest APIs
    - name: "datahub-gms-routes"
      match:
        - uri:
            prefix: /gms/
      rewrite:
        uri: /
      route:
        - destination:
            host: datahub-datahub-gms.datahub.svc.cluster.local
            port:
              number: 443
          headers:
            response:
              set:
                content-type: "application/json"
    {{- if .Values.global.datahub_standalone_consumers_enabled }}
    # Single endpoint used for checking functionalities for the deployed version
    - name: "datahub-mae-consumer-route"
      match:
        - uri:
            exact: /config
      route:
        - destination:
            host: datahub-datahub-mae-consumer.datahub.svc.cluster.local
            port:
              number: 9091
    {{- end }}
    {{- if .Values.dataHub.ui.enabled }}
    - name: "datahub-frontend-routes"
      match:
        - uri:
            prefix: /
      route:
        - destination:
            host: datahub-datahub-frontend.datahub.svc.cluster.local
            port:
              number: 443
    {{- end }}

Could you try running

Below is the response I’m getting.

{
  "models": {},
  "versions": {
    "linkedin/datahub": {
      "version": "v0.8.23",
      "commit": "a44b48a6b8a1df5f8d87330c5acc4957ef7d36b1"
    }
  },
  "statefulIngestionCapable": true,
  "retention": "true",
  "noCode": "true"
}

what about

curl --location --request POST '<http://datahub-datahub-gms:8080/entities?action=searchAcrossEntities>' \
--header 'X-RestLi-Protocol-Version: 2.0.0' \
--header 'Content-Type: application/json' \
--data-raw '{
    "entities": ["dataset"],
    "input": "*",
    "start": 0,
    "count": 1000
}'

Above curl returns a list of datasets from frontend pod

{"value":{"numEntities":10000,"pageSize":1000,"metadata":{"aggregations":[{"name":"entity","filterValues":[{"value":"dataset","facetCount":10000}],"aggregations":{"dataset":10000},"displayName":"Type"},{"name":"platform","filterValues":[{"value":"urn:li:dataPlatform:glue","facetCount":8440,"entity":"urn:li:dataPlatform:glue"},
....

Sorry about the delay. Could you try deploying v0.8.28 directly? We didn’t make any backward incompatible changes so far, but going backwards may have changed how we store the data

I will try it out and share how it goes. Thanks!

Hi @gray-carpet-60705, we are running into the same issue, I wonder if you were able to find out any solution to it?

Hi @bulky-grass-52762, unfortunately we didn't get to continue with this work.

Thanks for getting me back!