I m getting the following errors on my GMS pod logs Any idea DataHub #all-things-deployment

I’m getting the following errors on my GMS pod log...

rapid-house-76230

08/03/2022, 6:02 PM

I’m getting the following errors on my GMS pod logs. Any idea to how do I get more info about what these are?

Copy code

2022/08/03 21:35:16 Connected to <tcp://boot-foobar.c1.kafka-serverless.us-west-2.amazonaws.com:9098>
2022/08/03 21:35:16 Connected to <tcp://datahub.cluster-foobar.us-west-2.rds.amazonaws.com:5432>
2022/08/03 21:35:16 Received 200 from <https://datahub:Datahub%21123@search-datahub-elasticsearch-foobar.us-west-2.es.amazonaws.com:443>
2022/08/03 21:35:17 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:18 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:19 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:20 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:21 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:22 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:23 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:24 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 21:35:25 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s

I have the envs variables on my GMS deployment as follows

Copy code

- KAFKA_SCHEMAREGISTRY_URL=<http://schema-registry:8081>
      - KAFKA_BOOTSTRAP_SERVER=<http://boot-foobar.kafka-serverless.us-west-2.amazonaws.com:9098|boot-foobar.kafka-serverless.us-west-2.amazonaws.com:9098>
      - ALLOW_PLAINTEXT_LISTENER=yes
      - ZOOKEEPER_CLIENT_PORT=2181
      - ZOOKEEPER_TICK_TIME=2000
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=<PLAINTEXT://broker:29092>,PLAINTEXT_<HOST://localhost:9092>
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
      - KAFKA_HEAP_OPTS=-Xms256m -Xmx256m

Is there any config error or anything that you could think of on my end?

rapid-house-76230

08/03/2022, 10:24 PM

one interesting thing from the log is

Copy code

2022/08/03 22:20:42 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/08/03 22:20:43 Timeout after 4m0s waiting on dependencies to become available: [<tcp://datahub.cluster-foobar.us-west-2.rds.amazonaws.com:5432> <tcp://boot-foobar.c1.kafka-serverless.us-west-2.amazonaws.com:9098> <https://datahub:foobar@search-datahub-elasticsearch-foobar.us-west-2.es.amazonaws.com:443> http://]

rapid-house-76230

08/03/2022, 10:25 PM

does anyone know what the last http:// is for?

billions-morning-53195

08/04/2022, 8:42 PM

Set this value to

elasticsearch

? - https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L104

billions-morning-53195

08/04/2022, 8:44 PM

This empty value is looking for this EBEAN values from line 2-6, thats when I saw this error - https://github.com/datahub-project/datahub/blob/master/docker/datahub-gms/env/docker-without-neo4j.env

incalculable-ocean-74010

08/04/2022, 11:41 PM

Hello Tim, What version of datahub are you running?

rapid-house-76230

08/05/2022, 4:57 PM

@billions-morning-53195 the

graph_service_impl

flag was set to elasticsearch and all ebean vars were set, it was able to connect to the Postgres DB actually @incalculable-ocean-74010 the docker images we’re pulling for our k8s deployment all have :latest tags

incalculable-ocean-74010

08/05/2022, 7:25 PM

Latest as of when Tim? Latest is a a mutable tag. I would suggest moving to a static tag, I.e: https://hub.docker.com/layers/datahub-gms/linkedin/datahub-gms/cfcc1cf/images/sha256-be21055be86223e4a33b7d5e3bc59f2481c88a448de5dcff1c9ec2ba5259763b?context=explore

incalculable-ocean-74010

08/05/2022, 7:26 PM

Which is the latest gms image but on an immutable tag

rapid-house-76230

08/05/2022, 9:38 PM

Thanks Pedro, I just tried updating all images with :v0.8.42 tag, still having this issue though unfortunately

incalculable-ocean-74010

08/05/2022, 10:19 PM

Please use the tag I sent you. I believe v0.8.42 still has this bug

rapid-house-76230

08/05/2022, 10:21 PM

so do you suggest doing that only to GMS? Or would I have to find the latest tag for other images too, eg:

for frontend? https://hub.docker.com/layers/datahub-frontend-react/linkedin/datahub-frontend-react[…]b98c03b9cb7b2bc0c2f15aacdf4511da37a9422459694?context=explore

incalculable-ocean-74010

08/05/2022, 10:53 PM

For all images

rapid-house-76230

08/05/2022, 11:10 PM

Thanks Pedro, I updated all the tags, getting these now from GMS pod

Copy code

23:10:19.132 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mce-consumer-job-client-2, groupId=generic-mce-consumer-job-client] Error while fetching metadata with correlation id 792 : {MetadataChangeProposal_v1=UNKNOWN_TOPIC_OR_PARTITION}
23:10:19.132 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mce-consumer-job-client-2, groupId=generic-mce-consumer-job-client] Error while fetching metadata with correlation id 792 : {MetadataChangeProposal_v1=UNKNOWN_TOPIC_OR_PARTITION}
23:10:19.137 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mae-consumer-job-client-4, groupId=generic-mae-consumer-job-client] Error while fetching metadata with correlation id 783 : {MetadataChangeLog_Timeseries_v1=UNKNOWN_TOPIC_OR_PARTITION, MetadataChangeLog_Versioned_v1=UNKNOWN_TOPIC_OR_PARTITION}
23:10:19.137 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mae-consumer-job-client-4, groupId=generic-mae-consumer-job-client] Error while fetching metadata with correlation id 783 : {MetadataChangeLog_Timeseries_v1=UNKNOWN_TOPIC_OR_PARTITION, MetadataChangeLog_Versioned_v1=UNKNOWN_TOPIC_OR_PARTITION}
23:10:19.195 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-mce-consumer-job-client-1, groupId=mce-consumer-job-client] Error while fetching metadata with correlation id 780 : {MetadataChangeEvent_v4=UNKNOWN_TOPIC_OR_PARTITION}
23:10:19.195 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-mce-consumer-job-client-1, groupId=mce-consumer-job-client] Error while fetching metadata with correlation id 780 : {MetadataChangeEvent_v4=UNKNOWN_TOPIC_OR_PARTITION}

rapid-house-76230

08/05/2022, 11:12 PM

and still getting these errors from the frontend pod when I tried to log in (presumably because the gms pod wasn’t ready?)

Copy code

at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:331)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238)
	at java.lang.Thread.run(Thread.java:748)
23:11:23 [application-akka.actor.default-dispatcher-10] INFO  org.eclipse.jetty.util.log - Logging initialized @173177ms to org.eclipse.jetty.util.log.Slf4jLog
23:11:23 [application-akka.actor.default-dispatcher-10] WARN  o.e.j.j.spi.PropertyFileLoginModule - Exception starting propertyUserStore /etc/datahub/plugins/frontend/auth/user.props 
23:11:23 [application-akka.actor.default-dispatcher-10] ERROR application - The submitted callback is of type: class javax.security.auth.callback.NameCallback : javax.security.auth.callback.NameCallback@566f8c08
23:11:23 [application-akka.actor.default-dispatcher-10] ERROR application - The submitted callback is of type: class org.eclipse.jetty.jaas.callback.ObjectCallback : org.eclipse.jetty.jaas.callback.ObjectCallback@38848b4b
23:11:23 [application-akka.actor.default-dispatcher-10] WARN  application - The submitted callback is unsupported! 
23:11:23 [application-akka.actor.default-dispatcher-10] ERROR application - The submitted callback is of type: class javax.security.auth.callback.PasswordCallback : javax.security.auth.callback.PasswordCallback@3fe1f4dc
23:11:23 [application-akka.actor.default-dispatcher-10] ERROR application - The submitted callback is of type: class javax.security.auth.callback.NameCallback : javax.security.auth.callback.NameCallback@7a0c0b0f
23:11:23 [application-akka.actor.default-dispatcher-10] ERROR application - The submitted callback is of type: class org.eclipse.jetty.jaas.callback.ObjectCallback : org.eclipse.jetty.jaas.callback.ObjectCallback@5d79fcc5
23:11:23 [application-akka.actor.default-dispatcher-10] WARN  application - The submitted callback is unsupported! 
23:11:23 [application-akka.actor.default-dispatcher-10] ERROR application - The submitted callback is of type: class javax.security.auth.callback.PasswordCallback : javax.security.auth.callback.PasswordCallback@71829ced
23:11:23 [application-akka.actor.default-dispatcher-10] ERROR application - 

! @7ogkh7mop - Internal server error, for (POST) [/logIn] ->
 
play.api.UnexpectedException: Unexpected exception[RuntimeException: Failed to generate session token for user]
	at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:340)
	at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:263)
	at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:443)
	at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:441)
	at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)

incalculable-ocean-74010

08/08/2022, 8:43 AM

Hello Tim, How are you deploying DataHub? Via K8s? If so, what is the output of

kubectl -n <namespace where dathub is deployed> get po

incalculable-ocean-74010

08/08/2022, 8:44 AM

Is every pod ready? From a couple of messages ago it looked like GMS did not have elasticsearch environment variables. ElasticSearch is required for DataHub. You should have something like:

Copy code

ELASTICSEARCH_HOST=<http://vpc-datahub-pedro1-jdweenrudnvm4xvcnlf6hl5fky.us-west-2.es.amazonaws.com|vpc-datahub-pedro1-jdweenrudnvm4xvcnlf6hl5fky.us-west-2.es.amazonaws.com>
ELASTICSEARCH_USERNAME=admin
ELASTICSEARCH_USE_SSL=true
ELASTICSEARCH_PASSWORD=<password>
ELASTICSEARCH_PORT=443

incalculable-ocean-74010

08/08/2022, 8:44 AM

I’m using ElasticSearch managed service from AWS (open-search)

rapid-house-76230

08/08/2022, 5:49 PM

Yes! The AWS ES instance was up and connected successfully with my GMS. Here’s the output of

get po

Copy code

datahub-acryl-datahub-actions-5d97f7746d-vsllc     0/1     CrashLoopBackOff   1715       10d
datahub-frontend-deployment-54695d9dfc-5n7ql       1/1     Running            0          2d18h
datahub-gms-deployment-6c9d97df77-6k8g9            1/1     Running            0          2d18h
datahub-mae-consumer-deployment-547bf6d7ff-qc9p5   0/1     CrashLoopBackOff   781        2d19h
datahub-mce-consumer-deployment-77b7fc77-kx9cs     1/1     Running            2          2d19h
elasticsearch-master-0                             1/1     Running            0          10d
elasticsearch-master-1                             1/1     Running            0          10d
elasticsearch-master-2                             1/1     Running            0          10d
prerequisites-cp-schema-registry-cf79bfccf-m76bz   2/2     Running            0          10d
prerequisites-kafka-0                              1/1     Running            0          10d
prerequisites-mysql-0                              1/1     Running            0          10d
prerequisites-neo4j-community-0                    1/1     Running            0          10d
prerequisites-zookeeper-0                          1/1     Running            0          10d

rapid-house-76230

08/08/2022, 5:50 PM

GMS isn’t crashing but giving out these warnings

Copy code

17:50:09.106 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-mce-consumer-job-client-1, groupId=mce-consumer-job-client] Error while fetching metadata with correlation id 2407498 : {MetadataChangeEvent_v4=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.138 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-platform-event-job-client-5, groupId=generic-platform-event-job-client] Error while fetching metadata with correlation id 2408019 : {PlatformEvent_v1=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.138 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-platform-event-job-client-5, groupId=generic-platform-event-job-client] Error while fetching metadata with correlation id 2408019 : {PlatformEvent_v1=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.142 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mce-consumer-job-client-2, groupId=generic-mce-consumer-job-client] Error while fetching metadata with correlation id 2417041 : {MetadataChangeProposal_v1=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.142 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mce-consumer-job-client-2, groupId=generic-mce-consumer-job-client] Error while fetching metadata with correlation id 2417041 : {MetadataChangeProposal_v1=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.183 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mae-consumer-job-client-4, groupId=generic-mae-consumer-job-client] Error while fetching metadata with correlation id 2406373 : {MetadataChangeLog_Timeseries_v1=UNKNOWN_TOPIC_OR_PARTITION, MetadataChangeLog_Versioned_v1=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.183 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-generic-mae-consumer-job-client-4, groupId=generic-mae-consumer-job-client] Error while fetching metadata with correlation id 2406373 : {MetadataChangeLog_Timeseries_v1=UNKNOWN_TOPIC_OR_PARTITION, MetadataChangeLog_Versioned_v1=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.187 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-datahub-usage-event-consumer-job-client-3, groupId=datahub-usage-event-consumer-job-client] Error while fetching metadata with correlation id 2411122 : {DataHubUsageEvent_v1=UNKNOWN_TOPIC_OR_PARTITION}
17:50:09.187 [ThreadPoolTaskExecutor-1] WARN  o.apache.kafka.clients.NetworkClient:1077 - [Consumer clientId=consumer-datahub-usage-event-consumer-job-client-3, groupId=datahub-usage-event-consumer-job-client] Error while fetching metadata with correlation id 2411122 : {DataHubUsageEvent_v1=UNKNOWN_TOPIC_OR_PARTITION

rapid-house-76230

08/08/2022, 5:52 PM

The way I determine if things are working or not is to port forwarding the frontend pod and try to log in with datahub/dathub, and as of right now I’m still getting these from the frontend pod https://datahubspace.slack.com/archives/CV2UVAPPG/p1659741156232929?thread_ts=1659549753.745729&cid=CV2UVAPPG

incalculable-ocean-74010

08/09/2022, 9:06 AM

Can you ssh into the gms pod and try to ping the kafka cluster? Are you able to do so?

rapid-house-76230

08/09/2022, 4:37 PM

This issue was resolved by updating my

DATAHUB_GMS_HOST

datahub-gms-service

instead of

datahub-gms

(due to my k8s yaml config). Thanks @incalculable-ocean-74010!

4 Views

Open in Slack

Previous Next