https://datahubproject.io logo
Join Slack
Powered by
I ran "datahub docker quickstart" and it keeps res...
# troubleshoot
r
I ran "datahub docker quickstart" and it keeps restarting the mysql-setup, elasticsearch-setup and kafka-setup. Also the datahub-gms is unhealthy, I can acess the UI login page but I can't login. 
datahub docker check
says "datahub-gms is running but not healthy" Any thoughts?
Copy code
+ echo
+ grep -q ://
+ NEO4J_HOST=http://
+ [[ ! -z '' ]]
+ [[ -z '' ]]
+ ELASTICSEARCH_AUTH_HEADER='Accept: /'
+ [[ '' == true ]]
+ ELASTICSEARCH_PROTOCOL=http
+ WAIT_FOR_EBEAN=
+ [[ '' != true ]]
+ WAIT_FOR_EBEAN=' -wait <tcp://mysql:3306> '
+ WAIT_FOR_KAFKA=
+ [[ '' != true ]]
++ echo broker:29092
++ sed 's/,/ -wait tcp:\/\//g'
+ WAIT_FOR_KAFKA=' -wait <tcp://broker:29092> '
+ WAIT_FOR_NEO4J=
+ [[ elasticsearch != elasticsearch ]]
+ OTEL_AGENT=
+ [[ '' == true ]]
+ PROMETHEUS_AGENT=
+ [[ '' == true ]]
+ COMMON='
     -wait <tcp://mysql:3306>       -wait <tcp://broker:29092>           -timeout 240s     java -Xms1g -Xmx1g                -jar /jetty-runner.jar     --jar jetty-util.jar     --jar jetty-jmx.jar     --config /datahub/datahub-gms/scripts/jetty.xml     /datahub/datahub-gms/bin/war.war'
+ [[ '' != true ]]
+ exec dockerize -wait <http://elasticsearch:9200> -wait-http-header 'Accept: /' -wait <tcp://mysql:3306> -wait <tcp://broker:29092> -timeout 240s java -Xms1g -Xmx1g -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
2022/03/31 14:05:48 Waiting for: <http://elasticsearch:9200>
2022/03/31 14:05:48 Waiting for: <tcp://mysql:3306>
2022/03/31 14:05:48 Waiting for: <tcp://broker:29092>
2022/03/31 14:05:48 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:48 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:48 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:49 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:49 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:49 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:50 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:50 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:50 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:51 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:51 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:51 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:52 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:52 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:52 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:53 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:53 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:53 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:54 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:54 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:05:54 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:55 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:55 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:55 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:56 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:56 Problem with dial: dial tcp 172.19.0.6:29092: connect: connection refused. Sleeping 1s
2022/03/31 14:05:56 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:57 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:57 Connected to <tcp://broker:29092>
2022/03/31 14:05:57 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:58 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:58 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:05:59 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.19.0.3:9200: connect: connection refused. Sleeping 1s
2022/03/31 14:05:59 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:00 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:00 Received 200 from <http://elasticsearch:9200>
2022/03/31 14:06:01 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:02 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:03 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:04 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:05 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:06 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:07 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:08 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:09 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:10 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:11 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:12 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:13 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:14 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:15 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:16 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:17 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 14:06:19 Problem with dial: dial tcp 172.19.0.2:3306: connect: connection refused. Sleeping 1s
2022/03/31 14:06:20 Connected to <tcp://mysql:3306>
2022-03-31 14:06:20.702:INFO::main: Logging initialized @289ms to org.eclipse.jetty.util.log.StdErrLog
WARNING: jetty-runner is deprecated.
         See Jetty Documentation for startup options
         <https://www.eclipse.org/jetty/documentation/>
2022-03-31 14:06:20.731:INFO:oejr.Runner:main: Runner
2022-03-31 14:06:20.863:INFO:oeju.TypeUtil:main: JVM Runtime does not support Modules
2022-03-31 14:06:20.989:INFO:oejs.Server:main: jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_302-b08
2022-03-31 14:06:21.547:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@50cbc42f{/,null,UNAVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
java.util.zip.ZipException: invalid entry CRC (expected 0xe78b2198 but got 0xde56297a)
        at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:394)
        at java.util.zip.ZipInputStream.read(ZipInputStream.java:196)
        at java.util.jar.JarInputStream.read(JarInputStream.java:207)
        at org.eclipse.jetty.util.IO.copy(IO.java:172)
        at org.eclipse.jetty.util.IO.copy(IO.java:122)
        at org.eclipse.jetty.util.resource.JarResource.copyTo(JarResource.java:218)
        at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:636)
        at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:140)
        at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:488)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:523)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:407)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.server.Server.doStart(Server.java:371)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
        at org.eclipse.jetty.runner.Runner.main(Runner.java:565)
2022-03-31 14:06:21.577:INFO:oejs.AbstractConnector:main: Started ServerConnector@39a054a5{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2022-03-31 14:06:21.578:INFO:oejs.Server:main: Started @1168ms
datahub-gms logs
i
Hello Diogo, Could you run 
datahub docker nuke
& 
docker system prune --volumes
and try again?
It looks like the gms zip file got corrupted somehow.
r
Copy code
Fetching docker-compose file <https://raw.githubusercontent.com/linkedin/datahub/master/docker/quickstart/docker-compose-without-neo4j.quickstart.yml> from GitHub
Pulling elasticsearch          ... done
Pulling elasticsearch-setup    ... done
Pulling mysql                  ... done
Pulling datahub-gms            ... done
Pulling datahub-frontend-react ... done
Pulling datahub-actions        ... done
Pulling mysql-setup            ... done
Pulling zookeeper              ... done
Pulling broker                 ... done
Pulling schema-registry        ... done
Pulling kafka-setup            ... done

Creating network "datahub_network" with the default driver
Creating volume "datahub_esdata" with default driver
Creating volume "datahub_mysqldata" with default driver
Creating volume "datahub_zkdata" with default driver
Creating zookeeper           ... done
Creating mysql               ... done
Creating elasticsearch ... done
Creating elasticsearch-setup ... done
Creating mysql-setup         ... done
Creating datahub-gms         ... done
Creating broker                    ... done
Creating datahub_datahub-actions_1 ... done
Creating datahub-frontend-react    ... done
Creating schema-registry           ... done
Creating kafka-setup               ... done
.............
zookeeper is up-to-date
Starting mysql ...
elasticsearch is up-to-date
broker is up-to-date
Starting elasticsearch-setup ...
Starting mysql               ... done
kafka-setup is up-to-date
mysql-setup is up-to-date
Starting elasticsearch-setup ... done
datahub-frontend-react is up-to-date
datahub_datahub-actions_1 is up-to-date
..............
zookeeper is up-to-date
mysql is up-to-date
elasticsearch is up-to-date
broker is up-to-date
Starting mysql-setup ...
datahub-gms is up-to-date
Starting elasticsearch-setup ...
schema-registry is up-to-date
Starting mysql-setup         ... done
Starting elasticsearch-setup ... done
Starting kafka-setup         ... done
.............
elasticsearch is up-to-date
mysql is up-to-date
zookeeper is up-to-date
Starting elasticsearch-setup ...
datahub-gms is up-to-date
Starting mysql-setup         ...
broker is up-to-date
datahub_datahub-actions_1 is up-to-date
Starting elasticsearch-setup ... done
Starting mysql-setup         ... done
Starting kafka-setup         ... done
..............
mysql is up-to-date
zookeeper is up-to-date
elasticsearch is up-to-date
Starting mysql-setup ...
datahub-gms is up-to-date
Starting elasticsearch-setup ...
broker is up-to-date
datahub_datahub-actions_1 is up-to-date
Starting mysql-setup         ... done
Starting elasticsearch-setup ... done
Starting kafka-setup         ... done
It keeps repeating the last block starting the three setups
In the meantime the gms is starting and when I abort this with ctrl+c it becomes unhealthy
i
Give the docker QuickStart 15m to finish
Don’t ctrl+c
r
@incalculable-ocean-74010 After approximately 6 minutes it said:
Copy code
Unable to run quickstart - the following issues were detected:
- datahub-gms is running but not healthy
b
Can you do docker container ls?
And then perhaps grab the logs from the DataHub-GMS container using docker logs <container-Id>
r
@big-carpet-38439
Copy code
+ echo
+ grep -q ://
+ NEO4J_HOST=http://
+ [[ ! -z '' ]]
+ [[ -z '' ]]
+ ELASTICSEARCH_AUTH_HEADER='Accept: */*'
+ [[ '' == true ]]
+ ELASTICSEARCH_PROTOCOL=http
+ WAIT_FOR_EBEAN=
+ [[ '' != true ]]
+ WAIT_FOR_EBEAN=' -wait <tcp://mysql:3306> '
+ WAIT_FOR_KAFKA=
+ [[ '' != true ]]
++ echo broker:29092
++ sed 's/,/ -wait tcp:\/\//g'
+ WAIT_FOR_KAFKA=' -wait <tcp://broker:29092> '
+ WAIT_FOR_NEO4J=
+ [[ elasticsearch != elasticsearch ]]
+ OTEL_AGENT=
+ [[ '' == true ]]
+ PROMETHEUS_AGENT=
+ [[ '' == true ]]
+ COMMON='
     -wait <tcp://mysql:3306>       -wait <tcp://broker:29092>           -timeout 240s     java -Xms1g -Xmx1g                -jar /jetty-runner.jar     --jar jetty-util.jar     --jar jetty-jmx.jar     --config /datahub/datahub-gms/scripts/jetty.xml     /datahub/datahub-gms/bin/war.war'
+ [[ '' != true ]]
+ exec dockerize -wait <http://elasticsearch:9200> -wait-http-header 'Accept: */*' -wait <tcp://mysql:3306> -wait <tcp://broker:29092> -timeout 240s java -Xms1g -Xmx1g -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
2022/03/31 15:15:13 Waiting for: <http://elasticsearch:9200>
2022/03/31 15:15:13 Waiting for: <tcp://mysql:3306>
2022/03/31 15:15:13 Waiting for: <tcp://broker:29092>
2022/03/31 15:15:13 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:13 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:13 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:14 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:14 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:14 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:15 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:15 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:15 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:16 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:16 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:16 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:17 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:17 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:17 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:18 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:18 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:18 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:19 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:19 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:19 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:20 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:20 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:20 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:21 Problem with dial: dial tcp 172.22.0.8:29092: connect: connection refused. Sleeping 1s
2022/03/31 15:15:21 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:21 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:22 Connected to <tcp://broker:29092>
2022/03/31 15:15:22 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:22 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:23 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:23 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:24 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:24 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:25 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.22.0.2:9200: connect: connection refused. Sleeping 1s
2022/03/31 15:15:25 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:26 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:26 Received 200 from <http://elasticsearch:9200>
2022/03/31 15:15:27 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:28 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:29 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:30 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:31 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:32 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:33 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:34 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:35 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:36 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:37 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:38 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:39 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:40 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:41 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:42 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: server misbehaving. Sleeping 1s
2022/03/31 15:15:43 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:44 Problem with dial: dial tcp 172.22.0.4:3306: connect: connection refused. Sleeping 1s
2022/03/31 15:15:45 Connected to <tcp://mysql:3306>
2022-03-31 15:15:45.868:INFO::main: Logging initialized @307ms to org.eclipse.jetty.util.log.StdErrLog
WARNING: jetty-runner is deprecated.
         See Jetty Documentation for startup options
         <https://www.eclipse.org/jetty/documentation/>
2022-03-31 15:15:45.897:INFO:oejr.Runner:main: Runner
2022-03-31 15:15:46.023:INFO:oeju.TypeUtil:main: JVM Runtime does not support Modules
2022-03-31 15:15:46.150:INFO:oejs.Server:main: jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_302-b08
2022-03-31 15:15:46.722:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@50cbc42f{/,null,UNAVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
java.util.zip.ZipException: invalid entry CRC (expected 0xe78b2198 but got 0xde56297a)
        at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:394)
        at java.util.zip.ZipInputStream.read(ZipInputStream.java:196)
        at java.util.jar.JarInputStream.read(JarInputStream.java:207)
        at org.eclipse.jetty.util.IO.copy(IO.java:172)
        at org.eclipse.jetty.util.IO.copy(IO.java:122)
        at org.eclipse.jetty.util.resource.JarResource.copyTo(JarResource.java:218)
        at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:636)
        at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:140)
        at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:488)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:523)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:407)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.server.Server.doStart(Server.java:371)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
        at org.eclipse.jetty.runner.Runner.main(Runner.java:565)
2022-03-31 15:15:46.751:INFO:oejs.AbstractConnector:main: Started ServerConnector@39a054a5{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2022-03-31 15:15:46.751:INFO:oejs.Server:main: Started @1194ms
Some minutes after datahub-gms went unhealthy the acryl-datahub-actions exited (1)
acryl-datahub-actions logs just says this repetitively:
Copy code
2022/03/31 15:39:14 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:15 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:16 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:17 Received 503 from <http://datahub-gms:8080/health>. Sleeping 1s
2022/03/31 15:39:17 Timeout after 4m0s waiting on dependencies to become available: [<http://datahub-gms:8080/health>]
b
Copy code
o.e.j.w.WebAppContext@50cbc42f{/,null,UNAVAILABLE}{file:///datahub/datahub-gms/bin/war.war}
java.util.zip.ZipException: invalid entry CRC (expected 0xe78b2198 but got 0xde56297a)
This i have not yet seen - what type of machine are you deploying on?
b
hi @big-carpet-38439....i tried to deploy the new version of datahub and suddenly got this same error on quickstart
Unable to run quickstart - the following issues were detected: - datahub-gms is running but not healthy
CentOS Linux release 7.9.2009 (Core)
@red-window-75368 did you mansge to fix this?
b
Hmm - let me try to reproduce this locally... Something with the jar must be off. @blue-boots-43993 Are you using the very latest version?
just
Copy code
datahub docker quickstart
b
yes, the latest version and with above mentioned command
datahub CLI version is 
acryl-datahub, version 0.8.31.6
and docker-compose file is version 2.3
same errors even with nuke option (but --keep-data flag)
r
@big-carpet-38439 @blue-boots-43993 @incalculable-ocean-74010 Does any of you have any updates on this issue? On my side the error persists.
b
Unfortunately not...spent several hours and then ended up pulling old images from previous versions...this is currently a huge blocker for us
i
@red-window-75368 What OS are you running the cli on?
r
@incalculable-ocean-74010 Ubuntu 20.04
i
Strange, I can not reproduce this 🤔
b
@incalculable-ocean-74010 I tried again but always the same error.....obviously it's error with war file
i
@blue-boots-43993 would it be possible for you to compile gms locally and use that to overwrite the image being pulled down?
I can show you how
b
sure, can try
r
@blue-boots-43993 any success?
b
no 😞
i
@red-window-75368 I’m currently working with Roko on finding out what commit introduced this issue. It looks to be something affecting CentOS VMs. Whenever you get back offline, please DM me so we can see if you have the same issue.
f
Hi @incalculable-ocean-74010 Currently having the same issue with Roko and Diogo here. I have the datahub cli version: 0.8.31 in Ubuntu 20.04.3 LTS running on Oracle VM. Please let me know if its possible to be updated on the matter. I had tried to nuke and prune it, but the result is still the same with Diogo. Thank you
i
Hello @future-lawyer-31772 I’ve tried to reproduce this issue locally on my machine and on a clean slate Ubuntu VM and was not able to reproduce… Diogo uses an Ubuntu VM, Roko, a CentOS VM and you an Oracle Vm. This makes me think there is something across all VMs mentioned here that deviate from the software I used to setup the VM. Here is the setup I used:
Copy code
sudo apt-get update && sudo apt-get install git
sudo apt-get remove docker docker-engine <http://docker.io|docker.io> containerd runc

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] <https://download.docker.com/linux/ubuntu> \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli <http://containerd.io|containerd.io>
sudo usermod -aG docker $USER

 sudo curl -L "<https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname> -s)-$(uname -m)" -o /usr/local/bin/docker-compose
 sudo chmod +x /usr/local/bin/docker-compose
Copy code
(Log out and back in)
Copy code
sudo apt-get install openjdk-8-jdk
apt install python3.8-venv
sudo apt install python3-pip

python3 -m venv datahub-env             # create the environment
source datahub-env/bin/activate         # activate the environment

# Requires Python 3.6+
python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip install --upgrade acryl-datahub
datahub version
After this I did 
datahub docker quickstart
and everything worked as expected. Could you try the equivalent commands for Oracle & CentOS respectively?
420 Views
Open in Slack
PreviousNext
Powered by