https://datahubproject.io logo
Join Slack
Channels
acryl-omnisend
advice-data-governance
advice-metadata-modeling
all-things-datahub-in-windows
all-things-deployment
announcements
authentication-authorization
chatter
column-level-lineage
contribute
contribute-datahub-blog
data-council-workshop-2023
datahub-soda-test
demo-slack-notifications
design-business-glossary
design-data-product-entity
design-data-quality
design-datahub-documentation
design-dataset-access-requests
design-dataset-joins
feature-requests
flyte-datahub-integration
getting-started
github-activities
help-
i18n-community-contribution
ingestion
integration-alteryx-datahub
integration-azure-datahub
integration-dagster-datahub
integration-databricks-datahub
integration-datastudio-datahub
integration-iceberg-datahub
integration-powerbi-datahub
integration-prefect-datahub
integration-protobuf
integration-tableau-datahub
integration-vertica-datahub
introduce-yourself
jobs
metadata-day22-hackathon
muti-tenant-deployment
office-hours
openapi
plugins
show-and-tell
talk-data-product-management
troubleshoot
ui
Powered by Linen
Title
r

red-window-75368

03/31/2022, 2:15 PM
I ran "datahub docker quickstart" and it keeps restarting the mysql-setup, elasticsearch-setup and kafka-setup. Also the datahub-gms is unhealthy, I can acess the UI login page but I can't login. 
datahub docker check
says "datahub-gms is running but not healthy" Any thoughts?
+ echo
+ grep -q ://
+ NEO4J_HOST=http://
+ [[ ! -z '' ]]
+ [[ -z '' ]]
+ ELASTICSEARCH_AUTH_HEADER='Accept: /'
+ [[ '' == true ]]
+ ELASTICSEARCH_PROTOCOL=http
+ WAIT_FOR_EBEAN=
+ [[ '' != true ]]
+ WAIT_FOR_EBEAN=' -wait <tcp://mysql:3306> '
+ WAIT_FOR_KAFKA=
+ [[ '' != true ]]
++ echo broker:29092
++ sed 's/,/ -wait tcp:\/\//g'
+ WAIT_FOR_KAFKA=' -wait <tcp://broker:29092> '
+ WAIT_FOR_NEO4J=
+ [[ elasticsearch != elasticsearch ]]
+ OTEL_AGENT=
+ [[ '' == true ]]
+ PROMETHEUS_AGENT=
+ [[ '' == true ]]
+ COMMON='
     -wait <tcp://mysql:3306>       -wait <tcp://broker:29092>           -timeout 240s     java -Xms1g -Xmx1g                -jar /jetty-runner.jar     --jar jetty-util.jar     --jar jetty-jmx.jar     --config /datahub/datahub-gms/scripts/jetty.xml     /datahub/datahub-gms/bin/war.war'
+ [[ '' != true ]]
+ exec dockerize -wait <http://elasticsearch:9200> -wait-http-header 'Accept: /' -wait <tcp://mysql:3306> -wait <tcp://broker:29092> -timeout 240s java -Xms1g -Xmx1g -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
2022/03/31 14:05:48 Waiting for: <http://elasticsearch:9200>
2022/03/31 14:05:48 Waiting for: <tcp://mysql:3306>
2022/03/31 14:05:48 Waiting for: <tcp://broker:29092>
2022/03/31 14:05:48 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:48 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:48 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:49 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:49 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:49 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:50 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:50 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:50 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:51 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:51 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:51 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:52 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:52 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:52 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:53 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:53 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:53 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:54 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:54 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:54 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:55 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:55 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:55 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:56 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:56 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:56 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:57 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:57 Connected to <tcp://broker:29092>
2022/03/31 14:05:57 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:58 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:58 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:59 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:59 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:00 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:00 Received 200 from <http://elasticsearch:9200>
2022/03/31 14:06:01 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:02 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:03 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:04 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:05 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:06 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:07 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:08 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:09 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:10 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:11 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:12 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:13 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:14 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:15 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:16 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:17 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:19 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:06:20 Connected to <tcp://mysql:3306>
2022-03-31 14:06:20.702:INFO::main: Logging initialized @289ms to org.eclipse.jetty.util.log.StdErrLog
WARNING: jetty-runner is deprecated.
         See Jetty Documentation for startup options
         <https://www.eclipse.org/jetty/documentation/>
2022-03-31 14:06:20.731:INFO:oejr.Runner:main: Runner
2022-03-31 14:06:20.863:INFO:oeju.TypeUtil:main: JVM Runtime does not support Modules
2022-03-31 14:06:20.989:INFO:oejs.Server:main: jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_302-b08
2022-03-31 14:06:21.547:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@50cbc42f{/,null,UNAVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
java.util.zip.ZipException: invalid entry CRC (expected 0xe78b2198 but got 0xde56297a)
        at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:394)
        at java.util.zip.ZipInputStream.read(ZipInputStream.java:196)
        at java.util.jar.JarInputStream.read(JarInputStream.java:207)
        at org.eclipse.jetty.util.IO.copy(IO.java:172)
        at org.eclipse.jetty.util.IO.copy(IO.java:122)
        at org.eclipse.jetty.util.resource.JarResource.copyTo(JarResource.java:218)
        at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:636)
        at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:140)
        at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:488)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:523)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:407)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.server.Server.doStart(Server.java:371)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
        at org.eclipse.jetty.runner.Runner.main(Runner.java:565)
2022-03-31 14:06:21.577:INFO:oejs.AbstractConnector:main: Started ServerConnector@39a054a5{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2022-03-31 14:06:21.578:INFO:oejs.Server:main: Started @1168ms
datahub-gms logs
i

incalculable-ocean-74010

03/31/2022, 2:32 PM
Hello Diogo, Could you run 
datahub docker nuke
& 
docker system prune --volumes
and try again?
It looks like the gms zip file got corrupted somehow.
r

red-window-75368

03/31/2022, 2:38 PM
Fetching docker-compose file <https://raw.githubusercontent.com/linkedin/datahub/master/docker/quickstart/docker-compose-without-neo4j.quickstart.yml> from GitHub
Pulling elasticsearch          ... done
Pulling elasticsearch-setup    ... done
Pulling mysql                  ... done
Pulling datahub-gms            ... done
Pulling datahub-frontend-react ... done
Pulling datahub-actions        ... done
Pulling mysql-setup            ... done
Pulling zookeeper              ... done
Pulling broker                 ... done
Pulling schema-registry        ... done
Pulling kafka-setup            ... done

Creating network "datahub_network" with the default driver
Creating volume "datahub_esdata" with default driver
Creating volume "datahub_mysqldata" with default driver
Creating volume "datahub_zkdata" with default driver
Creating zookeeper           ... done
Creating mysql               ... done
Creating elasticsearch ... done
Creating elasticsearch-setup ... done
Creating mysql-setup         ... done
Creating datahub-gms         ... done
Creating broker                    ... done
Creating datahub_datahub-actions_1 ... done
Creating datahub-frontend-react    ... done
Creating schema-registry           ... done
Creating kafka-setup               ... done
.............
zookeeper is up-to-date
Starting mysql ...
elasticsearch is up-to-date
broker is up-to-date
Starting elasticsearch-setup ...
Starting mysql               ... done
kafka-setup is up-to-date
mysql-setup is up-to-date
Starting elasticsearch-setup ... done
datahub-frontend-react is up-to-date
datahub_datahub-actions_1 is up-to-date
..............
zookeeper is up-to-date
mysql is up-to-date
elasticsearch is up-to-date
broker is up-to-date
Starting mysql-setup ...
datahub-gms is up-to-date
Starting elasticsearch-setup ...
schema-registry is up-to-date
Starting mysql-setup         ... done
Starting elasticsearch-setup ... done
Starting kafka-setup         ... done
.............
elasticsearch is up-to-date
mysql is up-to-date
zookeeper is up-to-date
Starting elasticsearch-setup ...
datahub-gms is up-to-date
Starting mysql-setup         ...
broker is up-to-date
datahub_datahub-actions_1 is up-to-date
Starting elasticsearch-setup ... done
Starting mysql-setup         ... done
Starting kafka-setup         ... done
..............
mysql is up-to-date
zookeeper is up-to-date
elasticsearch is up-to-date
Starting mysql-setup ...
datahub-gms is up-to-date
Starting elasticsearch-setup ...
broker is up-to-date
datahub_datahub-actions_1 is up-to-date
Starting mysql-setup         ... done
Starting elasticsearch-setup ... done
Starting kafka-setup         ... done
It keeps repeating the last block starting the three setups
In the meantime the gms is starting and when I abort this with ctrl+c it becomes unhealthy
i

incalculable-ocean-74010

03/31/2022, 2:42 PM
Give the docker QuickStart 15m to finish
Don’t ctrl+c
r

red-window-75368

03/31/2022, 3:22 PM
@incalculable-ocean-74010 After approximately 6 minutes it said: 
Unable to run quickstart - the following issues were detected:
- datahub-gms is running but not healthy
b

big-carpet-38439

03/31/2022, 3:54 PM
Can you do docker container ls?
And then perhaps grab the logs from the DataHub-GMS container using docker logs <container-Id>
r

red-window-75368

03/31/2022, 3:57 PM
@big-carpet-38439
+ echo
+ grep -q ://
+ NEO4J_HOST=http://
+ [[ ! -z '' ]]
+ [[ -z '' ]]
+ ELASTICSEARCH_AUTH_HEADER='Accept: */*'
+ [[ '' == true ]]
+ ELASTICSEARCH_PROTOCOL=http
+ WAIT_FOR_EBEAN=
+ [[ '' != true ]]
+ WAIT_FOR_EBEAN=' -wait <tcp://mysql:3306> '
+ WAIT_FOR_KAFKA=
+ [[ '' != true ]]
++ echo broker:29092
++ sed 's/,/ -wait tcp:\/\//g'
+ WAIT_FOR_KAFKA=' -wait <tcp://broker:29092> '
+ WAIT_FOR_NEO4J=
+ [[ elasticsearch != elasticsearch ]]
+ OTEL_AGENT=
+ [[ '' == true ]]
+ PROMETHEUS_AGENT=
+ [[ '' == true ]]
+ COMMON='
     -wait <tcp://mysql:3306>       -wait <tcp://broker:29092>           -timeout 240s     java -Xms1g -Xmx1g                -jar /jetty-runner.jar     --jar jetty-util.jar     --jar jetty-jmx.jar     --config /datahub/datahub-gms/scripts/jetty.xml     /datahub/datahub-gms/bin/war.war'
+ [[ '' != true ]]
+ exec dockerize -wait <http://elasticsearch:9200> -wait-http-header 'Accept: */*' -wait <tcp://mysql:3306> -wait <tcp://broker:29092> -timeout 240s java -Xms1g -Xmx1g -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
2022/03/31 15:15:13 Waiting for: <http://elasticsearch:9200>
2022/03/31 15:15:13 Waiting for: <tcp://mysql:3306>
2022/03/31 15:15:13 Waiting for: <tcp://broker:29092>
2022/03/31 15:15:13 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:13 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:13 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:14 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:14 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:14 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:15 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:15 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:15 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:16 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:16 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:16 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:17 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:17 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:17 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:18 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:18 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:18 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:19 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:19 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:19 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:20 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:20 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:20 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:21 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:21 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:21 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:22 Connected to <tcp://broker:29092>
2022/03/31 15:15:22 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:22 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:23 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:23 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:24 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:24 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:25 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:25 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:26 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:26 Received 200 from <http://elasticsearch:9200>
2022/03/31 15:15:27 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:28 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:29 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:30 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:31 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:32 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:33 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:34 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:35 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:36 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:37 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:38 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:39 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:40 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:41 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:42 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:43 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:44 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:45 Connected to <tcp://mysql:3306>
2022-03-31 15:15:45.868:INFO::main: Logging initialized @307ms to org.eclipse.jetty.util.log.StdErrLog
WARNING: jetty-runner is deprecated.
         See Jetty Documentation for startup options
         <https://www.eclipse.org/jetty/documentation/>
2022-03-31 15:15:45.897:INFO:oejr.Runner:main: Runner
2022-03-31 15:15:46.023:INFO:oeju.TypeUtil:main: JVM Runtime does not support Modules
2022-03-31 15:15:46.150:INFO:oejs.Server:main: jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_302-b08
2022-03-31 15:15:46.722:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@50cbc42f{/,null,UNAVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
java.util.zip.ZipException: invalid entry CRC (expected 0xe78b2198 but got 0xde56297a)
        at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:394)
        at java.util.zip.ZipInputStream.read(ZipInputStream.java:196)
        at java.util.jar.JarInputStream.read(JarInputStream.java:207)
        at org.eclipse.jetty.util.IO.copy(IO.java:172)
        at org.eclipse.jetty.util.IO.copy(IO.java:122)
        at org.eclipse.jetty.util.resource.JarResource.copyTo(JarResource.java:218)
        at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:636)
        at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:140)
        at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:488)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:523)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:407)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.server.Server.doStart(Server.java:371)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
        at org.eclipse.jetty.runner.Runner.main(Runner.java:565)
2022-03-31 15:15:46.751:INFO:oejs.AbstractConnector:main: Started ServerConnector@39a054a5{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2022-03-31 15:15:46.751:INFO:oejs.Server:main: Started @1194ms
Some minutes after datahub-gms went unhealthy the acryl-datahub-actions exited (1)
acryl-datahub-actions logs just says this repetitively: 
2022/03/31 15:39:14 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:15 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:16 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:17 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:17 Timeout after 4m0s waiting on dependencies to become available: [<http://datahub-gms:8080/health>]
b

big-carpet-38439

03/31/2022, 6:14 PM
o.e.j.w.WebAppContext@50cbc42f{/,null,UNAVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
java.util.zip.ZipException: invalid entry CRC (expected 0xe78b2198 but got 0xde56297a)
This i have not yet seen - what type of machine are you deploying on?
b

blue-boots-43993

04/01/2022, 4:46 PM
hi @big-carpet-38439....i tried to deploy the new version of datahub and suddenly got this same error on quickstart
Unable to run quickstart - the following issues were detected: - datahub-gms is running but not healthy
CentOS Linux release 7.9.2009 (Core)
@red-window-75368 did you mansge to fix this?
b

big-carpet-38439

04/01/2022, 9:14 PM
Hmm - let me try to reproduce this locally... Something with the jar must be off. @blue-boots-43993 Are you using the very latest version?
just 
datahub docker quickstart
b

blue-boots-43993

04/02/2022, 8:05 AM
yes, the latest version and with above mentioned command
datahub CLI version is 
acryl-datahub, version 0.8.31.6
and docker-compose file is version 2.3
same errors even with nuke option (but --keep-data flag)
r

red-window-75368

04/04/2022, 11:03 AM
@big-carpet-38439 @blue-boots-43993 @incalculable-ocean-74010 Does any of you have any updates on this issue? On my side the error persists.
b

blue-boots-43993

04/04/2022, 11:04 AM
Unfortunately not...spent several hours and then ended up pulling old images from previous versions...this is currently a huge blocker for us
i

incalculable-ocean-74010

04/04/2022, 11:22 AM
@red-window-75368 What OS are you running the cli on?
r

red-window-75368

04/04/2022, 12:40 PM
@incalculable-ocean-74010 Ubuntu 20.04
i

incalculable-ocean-74010

04/04/2022, 2:00 PM
Strange, I can not reproduce this 🤔
b

blue-boots-43993

04/04/2022, 4:34 PM
@incalculable-ocean-74010 I tried again but always the same error.....obviously it's error with war file
i

incalculable-ocean-74010

04/04/2022, 4:37 PM
@blue-boots-43993 would it be possible for you to compile gms locally and use that to overwrite the image being pulled down?
I can show you how
b

blue-boots-43993

04/04/2022, 4:37 PM
sure, can try
r

red-window-75368

04/05/2022, 7:55 AM
@blue-boots-43993 any success?
b

blue-boots-43993

04/05/2022, 7:55 AM
no 😞
i

incalculable-ocean-74010

04/05/2022, 6:25 PM
@red-window-75368 I’m currently working with Roko on finding out what commit introduced this issue. It looks to be something affecting CentOS VMs. Whenever you get back offline, please DM me so we can see if you have the same issue.
f

future-lawyer-31772

04/07/2022, 3:36 AM
Hi @incalculable-ocean-74010 Currently having the same issue with Roko and Diogo here. I have the datahub cli version: 0.8.31 in Ubuntu 20.04.3 LTS running on Oracle VM. Please let me know if its possible to be updated on the matter. I had tried to nuke and prune it, but the result is still the same with Diogo. Thank you
i

incalculable-ocean-74010

04/07/2022, 8:44 AM
Hello @future-lawyer-31772 I’ve tried to reproduce this issue locally on my machine and on a clean slate Ubuntu VM and was not able to reproduce… Diogo uses an Ubuntu VM, Roko, a CentOS VM and you an Oracle Vm. This makes me think there is something across all VMs mentioned here that deviate from the software I used to setup the VM. Here is the setup I used: 
sudo apt-get update && sudo apt-get install git
sudo apt-get remove docker docker-engine <http://docker.io|docker.io> containerd runc

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] <https://download.docker.com/linux/ubuntu> \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli <http://containerd.io|containerd.io>
sudo usermod -aG docker $USER

 sudo curl -L "<https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname> -s)-$(uname -m)" -o /usr/local/bin/docker-compose
 sudo chmod +x /usr/local/bin/docker-compose
(Log out and back in)
sudo apt-get install openjdk-8-jdk
apt install python3.8-venv
sudo apt install python3-pip

python3 -m venv datahub-env             # create the environment
source datahub-env/bin/activate         # activate the environment

# Requires Python 3.6+
python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip install --upgrade acryl-datahub
datahub version
After this I did 
datahub docker quickstart
and everything worked as expected. Could you try the equivalent commands for Oracle & CentOS respectively?
View count: 353
Back to #troubleshootJoin thread in Slack