Hello, I’m trying to upgrade DataHub from v0.8.21 ...
# troubleshoot
g
Hello, I’m trying to upgrade DataHub from v0.8.21 to v0.8.23 (and eventually to the latest) and running into UI login error as default “datahub” user upon initial upgrade. I’ll include more details in the thread. Any help would be greatly appreciated. Thanks!
Login error screenshot and error log from frontend are attached.
Copy code
20:21:53 [main] INFO  o.p.play.store.ShiroAesDataEncrypter - Generated key: <key>
20:21:55 [application-akka.actor.default-dispatcher-2] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
20:21:55 [application-akka.actor.default-dispatcher-3] WARN  akka.util.ManifestInfo - Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.23] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.5.23, 2.5.11] in libraries [akka-protobuf:2.5.23, akka-actor:2.5.23, akka-slf4j:2.5.11, akka-stream:2.5.23]
[WARN] [03/14/2022 20:21:55.409] [main] [ManifestInfo(<akka://proxyClient>)] Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.23] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.5.23, 2.5.11] in libraries [akka-protobuf:2.5.23, akka-actor:2.5.23, akka-slf4j:2.5.11, akka-stream:2.5.23]
20:21:57 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
20:21:57 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No callback executor is provided to HttpClientFactory, using it's own call back executor.
20:21:57 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
20:21:57 [main] INFO  c.l.r.t.h.client.HttpClientFactory - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 '
20:21:57 [main] INFO  o.a.k.c.producer.ProducerConfig - ProducerConfig values:
	acks = 1
	batch.size = 16384
	bootstrap.servers = [<server1>:9094, <server2>:9094, <server3>:9094]
	buffer.memory = 33554432
	client.dns.lookup = default
	client.id = datahub-frontend
	compression.type = none
	<http://connections.max.idle.ms|connections.max.idle.ms> = 540000
	<http://delivery.timeout.ms|delivery.timeout.ms> = 120000
	enable.idempotence = false
	interceptor.classes = []
	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
	<http://linger.ms|linger.ms> = 0
	<http://max.block.ms|max.block.ms> = 60000
	max.in.flight.requests.per.connection = 5
	max.request.size = 1048576
	<http://metadata.max.age.ms|metadata.max.age.ms> = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	<http://metrics.sample.window.ms|metrics.sample.window.ms> = 30000
	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
	receive.buffer.bytes = 32768
	<http://reconnect.backoff.max.ms|reconnect.backoff.max.ms> = 1000
	<http://reconnect.backoff.ms|reconnect.backoff.ms> = 50
	<http://request.timeout.ms|request.timeout.ms> = 30000
	retries = 2147483647
	<http://retry.backoff.ms|retry.backoff.ms> = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = SSL
	send.buffer.bytes = 131072
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = [hidden]
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = /mnt/kafka/ssl/keystore
	ssl.keystore.password = [hidden]
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = /mnt/kafka/public/truststore
	ssl.truststore.password = [hidden]
	ssl.truststore.type = JKS
	<http://transaction.timeout.ms|transaction.timeout.ms> = 60000
	transactional.id = null
	value.serializer = class org.apache.kafka.common.serialization.StringSerializer
20:21:57 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0
20:21:57 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId: fc1aaa116b661c8a
20:21:57 [main] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1647289317692
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
20:21:58 [kafka-producer-network-thread | datahub-frontend] INFO  org.apache.kafka.clients.Metadata - [Producer clientId=datahub-frontend] Cluster ID: nEDo3IP_Tm2wgUkpQN8RfQ
20:21:58 [main] INFO  play.api.Play - Application started (Prod)
20:21:59 [main] INFO  play.core.server.AkkaHttpServer - Listening for HTTP on /0.0.0.0:9002
20:25:16 [application-akka.actor.default-dispatcher-6] INFO  org.eclipse.jetty.util.log - Logging initialized @206589ms to org.eclipse.jetty.util.log.Slf4jLog
20:25:16 [application-akka.actor.default-dispatcher-6] WARN  application - The submitted callback is unsupported!
20:25:26 [application-akka.actor.default-dispatcher-6] ERROR application -
! @7n2chf04n - Internal server error, for (POST) [/logIn] ->

play.api.UnexpectedException: Unexpected exception[RuntimeException: Failed to generate session token for user]
	at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:247)
	at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:176)
	at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:363)
	at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:361)
	at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:346)
	at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:92)
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:92)
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:92)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.RuntimeException: Failed to generate session token for user
	at client.AuthServiceClient.generateSessionTokenForUser(AuthServiceClient.java:80)
	at controllers.AuthenticationController.logIn(AuthenticationController.java:139)
	at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$5$$anonfun$apply$5.apply(Routes.scala:380)
	at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$5$$anonfun$apply$5.apply(Routes.scala:380)
	at play.core.routing.HandlerInvokerFactory$$anon$3.resultCall(HandlerInvoker.scala:134)
	at play.core.routing.HandlerInvokerFactory$$anon$3.resultCall(HandlerInvoker.scala:133)
	at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$8$$anon$2$$anon$1.invocation(HandlerInvoker.scala:108)
	at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:88)
	at play.http.DefaultActionCreator$1.call(DefaultActionCreator.java:31)
	at play.core.j.JavaAction$$anonfun$9.apply(JavaAction.scala:138)
	at play.core.j.JavaAction$$anonfun$9.apply(JavaAction.scala:138)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
	at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:56)
	at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:70)
	at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:48)
	at scala.concurrent.impl.Future$.apply(Future.scala:31)
	at scala.concurrent.Future$.apply(Future.scala:494)
	at play.core.j.JavaAction.apply(JavaAction.scala:138)
	at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:96)
	at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:89)
	at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
	at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
	... 13 common frames omitted
Caused by: java.lang.RuntimeException: Bad response from the Metadata Service: HTTP/1.1 503 Service Unavailable ResponseEntityProxy{[Content-Type: text/plain,Content-Length: 91,Chunked: false]}
	at client.AuthServiceClient.generateSessionTokenForUser(AuthServiceClient.java:76)
	... 35 common frames omitted
b
This @gray-carpet-60705 - It appears that the backend (datahub-gms) is down. Can you confirm whether that container is up and running??
g
Hi @big-carpet-38439, thanks for the reply. gms and mce/mae consumers all show up as running, but they don’t appear to be responsive. No log is available for mce, and gms log stopped after the following. I’m installing Helm chart version 0.2.41 with image tag v0.8.23, and we are on AWS using EKS, RDS Postgres. MSK, Glue schema registry, and ES. Is there something else I can try to get gms back up?
Copy code
20:22:02.902 [main] INFO  c.a.s.s.c.c.GlueSchemaRegistryConfiguration:188 - Cache Time to live is not found, using default 86400000
20:22:04.107 [main] INFO  o.a.kafka.common.utils.AppInfoParser:117 - Kafka version: 5.5.1-ccs
20:22:04.108 [main] INFO  o.a.kafka.common.utils.AppInfoParser:118 - Kafka commitId: cb1873c1fdf5f5f9
20:22:04.108 [main] INFO  o.a.kafka.common.utils.AppInfoParser:119 - Kafka startTimeMs: 1647289324103
20:22:04.320 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - dataHubPolicyKey schema is compatible with previous schema due to
20:22:04.322 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - dataHubPolicyInfo schema is compatible with previous schema due to
20:22:04.322 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserKey schema is compatible with previous schema due to
20:22:04.323 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserEditableInfo schema is compatible with previous schema due to
20:22:04.323 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserInfo schema is compatible with previous schema due to
20:22:04.324 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - corpUserStatus schema is compatible with previous schema due to
20:22:04.324 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - dataJobKey schema is compatible with previous schema due to
20:22:04.325 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - datasetKey schema is compatible with previous schema due to
20:22:04.325 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - viewProperties schema is compatible with previous schema due to
20:22:04.328 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - schemaMetadata schema is compatible with previous schema due to
20:22:04.329 [main] INFO  c.l.m.m.r.MergedEntityRegistry:83 - status schema is compatible with previous schema due to
20:22:04.978 [kafka-producer-network-thread | producer-1] INFO  org.apache.kafka.clients.Metadata:277 - [Producer clientId=producer-1] Cluster ID: ...
20:22:04.978 [kafka-producer-network-thread | producer-1] INFO  org.apache.kafka.clients.Metadata:277 - [Producer clientId=producer-1] Cluster ID: ...
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8Mar 14, 2022 8:22:05 PM org.neo4j.driver.internal.logging.JULogger info
INFO: Direct driver instance 76314483 created for server address localhost:7687
20:22:05.630 [main] WARN  c.l.r.t.h.client.HttpClientFactory:894 - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
20:22:05.632 [main] WARN  c.l.r.t.h.client.HttpClientFactory:903 - No callback executor is provided to HttpClientFactory, using it's own call back executor.
20:22:05.633 [main] WARN  c.l.r.t.h.client.HttpClientFactory:911 - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
20:22:05.647 [main] INFO  c.l.r.t.h.client.HttpClientFactory:1338 - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 '
20:22:05.969 [main] WARN  c.l.r.t.h.client.HttpClientFactory:894 - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
20:22:05.969 [main] WARN  c.l.r.t.h.client.HttpClientFactory:903 - No callback executor is provided to HttpClientFactory, using it's own call back executor.
20:22:05.969 [main] WARN  c.l.r.t.h.client.HttpClientFactory:911 - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
20:22:05.970 [main] INFO  c.l.r.t.h.client.HttpClientFactory:1338 - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 '
20:22:06.495 [main] INFO  c.l.r.s.c.ResourceMethodConfigProviderImpl:75 - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
20:22:06.517 [main] INFO  c.l.r.s.c.ResourceMethodConfigProviderImpl:75 - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
20:22:06.519 [main] INFO  c.l.r.s.c.ResourceMethodConfigProviderImpl:75 - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
20:22:07.835 [main] INFO  c.l.g.f.k.SimpleKafkaConsumerFactory:47 - Simple KafkaListenerContainerFactory built successfully
20:22:07.856 [main] INFO  c.l.g.f.k.KafkaEventConsumerFactory:82 - Event-based KafkaListenerContainerFactory built successfully
20:22:07.859 [main] INFO  c.l.g.f.k.s.KafkaSchemaRegistryFactory:58 - creating schema registry config using url: <http://prerequisites-cp-schema-registry:8081>
20:22:07.912 [main] INFO  c.l.metadata.boot.BootstrapManager:24 - Starting Bootstrap Process...
20:22:07.912 [main] INFO  c.l.metadata.boot.BootstrapManager:28 - Executing bootstrap step 1/5 with name IngestRootUserStep...
20:22:08.684 [main] INFO  c.l.metadata.boot.BootstrapManager:28 - Executing bootstrap step 2/5 with name IngestPoliciesStep...
20:22:08.685 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:49 - Ingesting default access policies...
20:22:08.698 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:0
20:22:08.789 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:1
20:22:08.842 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:2
20:22:08.881 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:3
20:22:08.924 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:4
20:22:08.975 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:5
20:22:09.008 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:69 - Ingesting default policy with urn urn:li:dataHubPolicy:6
20:22:09.048 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:77 - Skipping ingestion of editable policy with urn urn:li:dataHubPolicy:7
20:22:09.048 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:81 - Successfully ingested default access policies.
20:22:09.049 [main] INFO  c.l.metadata.boot.BootstrapManager:28 - Executing bootstrap step 3/5 with name IngestDataPlatformsStep...
20:22:09.807 [main] INFO  c.l.metadata.boot.BootstrapManager:36 - Starting asynchronous bootstrap step 4/5 with name IngestDataPlatformInstancesStep...
20:22:09.812 [main] INFO  c.l.metadata.boot.BootstrapManager:36 - Starting asynchronous bootstrap step 5/5 with name IngestRetentionPoliciesStep...
20:22:09.812 [Thread-13] INFO  c.l.m.b.s.IngestDataPlatformInstancesStep:61 - Checking for DataPlatformInstance
20:22:09.813 [Thread-14] INFO  c.l.m.b.s.IngestRetentionPoliciesStep:47 - Ingesting default retention...
20:22:09.813 [Thread-14] INFO  c.l.m.b.s.IngestRetentionPoliciesStep:51 - IngestRetentionPolicies disabled. Skipping.
20:22:09.818 [Thread-13] INFO  c.l.m.b.s.IngestDataPlatformInstancesStep:67 - DataPlaformInstance aspect exists. Skipping step
20:22:09.822 [main] INFO  o.s.web.context.ContextLoader:307 - Root WebApplicationContext initialized in 16887 ms
20:22:09.858 [main] INFO  c.d.a.filter.AuthenticationFilter:143 - Auth is disabled. Building no-op authenticator chain...
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.82022-03-14 20:22:10.002:INFO:oejshC.ROOT:main: Initializing Spring DispatcherServlet 'apiServlet'
20:22:10.002 [main] INFO  o.s.web.servlet.DispatcherServlet:525 - Initializing Servlet 'apiServlet'
20:22:10.070 [main] ERROR c.datahub.graphql.GraphQLController:32 - I created graphqlcontroller
20:22:10.362 [main] WARN  o.s.h.c.j.Jackson2ObjectMapperBuilder:127 - For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
20:22:10.598 [main] INFO  o.s.web.servlet.DispatcherServlet:547 - Completed initialization in 596 ms
2022-03-14 20:22:10.599:INFO:oejshC.ROOT:main: Initializing Spring DispatcherServlet 'authServlet'
20:22:10.599 [main] INFO  o.s.web.servlet.DispatcherServlet:525 - Initializing Servlet 'authServlet'
20:22:10.673 [main] INFO  o.s.web.servlet.DispatcherServlet:547 - Completed initialization in 74 ms
2022-03-14 20:22:10.674:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@60215eee{Open source GMS,/,file:///tmp/jetty-0_0_0_0-8080-war_war-_-any-5633708735902109725.dir/webapp/,AVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
2022-03-14 20:22:10.747:INFO:oejs.AbstractConnector:main: Started ServerConnector@6ed3ef1{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2022-03-14 20:22:10.747:INFO:oejs.Server:main: Started @29727ms
b
Oh interesting.. These logs seem to indicate that GMS is indeed up!
Bringing in my friend Dexter 🙂 @early-lamp-41924 who may have seen this before
e
hmn seems like it started fine. wonder why frontend is unable to reach it. Can you try portforwarding
Copy code
kubectl port-forward svc/datahub-datahub-gms 8080:8080
Ah nvm you tried this and worked right?
g
This is the Istio gateway and virtual service that we have originally set up. Not sure if these rewrites are causing any problem.
Copy code
apiVersion: <http://networking.istio.io/v1alpha3|networking.istio.io/v1alpha3>
kind: Gateway
metadata:
  name: datahub-gateway
spec:
  selector:
    istio: datahub-ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: datahub-tls-secret
      hosts:
        - {{ .Values.dataHub.hostname | quote }}
---
apiVersion: <http://networking.istio.io/v1alpha3|networking.istio.io/v1alpha3>
kind: VirtualService
metadata:
  name: datahub-service
spec:
  hosts:
    - {{ .Values.dataHub.hostname | quote }}
  gateways:
    - datahub-gateway
  http:
    # Routes for gms ingest APIs
    - name: "datahub-gms-routes"
      match:
        - uri:
            prefix: /gms/
      rewrite:
        uri: /
      route:
        - destination:
            host: datahub-datahub-gms.datahub.svc.cluster.local
            port:
              number: 443
          headers:
            response:
              set:
                content-type: "application/json"
    {{- if .Values.global.datahub_standalone_consumers_enabled }}
    # Single endpoint used for checking functionalities for the deployed version
    - name: "datahub-mae-consumer-route"
      match:
        - uri:
            exact: /config
      route:
        - destination:
            host: datahub-datahub-mae-consumer.datahub.svc.cluster.local
            port:
              number: 9091
    {{- end }}
    {{- if .Values.dataHub.ui.enabled }}
    - name: "datahub-frontend-routes"
      match:
        - uri:
            prefix: /
      route:
        - destination:
            host: datahub-datahub-frontend.datahub.svc.cluster.local
            port:
              number: 443
    {{- end }}
Below is what I see currently with
/config
endpoint.
Copy code
{
  "versions": {
    "linkedin/datahub": {
      "version": "v0.8.23",
      "commit": "a44b48a6b8a1df5f8d87330c5acc4957ef7d36b1"
    }
  },
  "noCode": "true"
}
e
Could you try running
Copy code
kubectl exec -n <<namespace>> --stdin --tty <<frontend-pod-name>> -- /bin/sh
and then curl
g
Below is the response I’m getting.
Copy code
{
  "models": {},
  "versions": {
    "linkedin/datahub": {
      "version": "v0.8.23",
      "commit": "a44b48a6b8a1df5f8d87330c5acc4957ef7d36b1"
    }
  },
  "statefulIngestionCapable": true,
  "retention": "true",
  "noCode": "true"
}
e
what about
Copy code
curl --location --request POST '<http://datahub-datahub-gms:8080/entities?action=searchAcrossEntities>' \
--header 'X-RestLi-Protocol-Version: 2.0.0' \
--header 'Content-Type: application/json' \
--data-raw '{
    "entities": ["dataset"],
    "input": "*",
    "start": 0,
    "count": 1000
}'
@big-carpet-38439 seems like frontend is able to talk to gms fine
any other reason why this error might show up?
g
Above curl returns a list of datasets from frontend pod
Copy code
{"value":{"numEntities":10000,"pageSize":1000,"metadata":{"aggregations":[{"name":"entity","filterValues":[{"value":"dataset","facetCount":10000}],"aggregations":{"dataset":10000},"displayName":"Type"},{"name":"platform","filterValues":[{"value":"urn:li:dataPlatform:glue","facetCount":8440,"entity":"urn:li:dataPlatform:glue"},
....
Not sure if it is relevant, but I did try deploying the latest (v0.8.28) earlier once before deciding to upgrade incrementally.
e
Sorry about the delay. Could you try deploying v0.8.28 directly? We didn’t make any backward incompatible changes so far, but going backwards may have changed how we store the data
g
I will try it out and share how it goes. Thanks!
b
Hi @gray-carpet-60705, we are running into the same issue, I wonder if you were able to find out any solution to it?
g
Hi @bulky-grass-52762, unfortunately we didn't get to continue with this work.
thankyou 1
b
Thanks for getting me back!