https://datahubproject.io logo
Join Slack
Powered by
Hi all. I have version 9.0 of Data Hub deployed on...
# troubleshoot
c
Hi all. I have version 9.0 of Data Hub deployed on RHEL9 but I am having some connection issues. Below the issues (quickstart was unable to run) : - kafka-setup is still running - schema-registry is not running - broker is not running - zookeeper is not running - datahub-gms is still starting - mysql-setup did not exit cleanly Version details : DataHub CLI version: 0.9.0.4 Python version: 3.9.10 (main, Feb 9 2022, 000000) [GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] It seems that there some connexion errors : (...) datahub-datahub-actions-1 | 2022/10/27 065100 Problem with request: Get "http://datahub-gms:8080/health": dial tcp 172.18.0.58080 connect: connection refused. Sleeping 1s datahub-datahub-actions-1 | 2022/10/27 065101 Problem with request: Get "http://datahub-gms:8080/health": dial tcp 172.18.0.58080 connect: connection refused. Sleeping 1s (...) elasticsearch-setup | 2022/10/27 070522 Problem with request: Get http://elasticsearch:9200: dial tcp 172.18.0.29200 connect: connection refused. Sleeping 1s (...) datahub-gms | 2022/10/27 064926 Problem with dial: dial tcp: lookup broker on 127.0.0.1153 server misbehaving. Sleeping 1s (...) datahub-gms | 2022/10/27 064927 Problem with dial: dial tcp 172.18.0.929092 connect: connection refused. Sleeping 1s (...) datahub-frontend-react | sasl.kerberos.service.name = null datahub-gms | 2022/10/27 065055 Problem with dial: dial tcp 172.18.0.1129092 connect: connection refused. Sleeping 1s (...) Has anyone seen this before or has any advice of what I can troubleshoot to get Data Hub running properly? Any help much appreciated!
i
Hello Stefan, Can you share the logs for the GMS container? DataHub has some issues running on RHEL systems.
c
Hi Pedro, please find attached the log file
tmphvrfdyvo.log
i
For future reference please only share the gms logs, makes it much easier to parse the file. What version of docker are you using? What is your system’s architecture, x86 or ARM?
c
Here is the Docker Version I'm using : Docker version 20.10.21, build baeda1f And my system architecture : Linux frreet41 5.14.0-70.26.1.el9_0.x86_64 #1 SMP PREEMPT Fri Sep 2 160740 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
i
Can you do
Copy code
datahub docker nuke --keep-data
datahub docker quickstart --version v0.9.0
docker ps -a
One command at a time
And the version of the datahub cli tool?
datahub version
c
The Datahub version is : DataHub CLI version: 0.9.0.4 Python version: 3.9.10 (main, Feb 9 2022, 000000) [GCC 11.2.1 20220127 (Red Hat 11.2.1-9)]
Below the results : Unable to run quickstart - the following issues were detected: - kafka-setup is still running - schema-registry is not running - broker is not running - datahub-gms is still starting - mysql-setup did not exit cleanly - zookeeper is not running CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 94f0e98ca68f linkedin/datahub-kafka-setup:v0.9.0 "/bin/sh -c ./kafka-…" 6 minutes ago Up 6 minutes kafka-setup 3751e29d5a90 confluentinc/cp-schema-registry:5.4.0 "/etc/confluent/dock…" 6 minutes ago Exited (1) 54 seconds ago schema-registry 4692fe3a8468 acryldata/datahub-actions:head "/bin/sh -c 'dockeri…" 6 minutes ago Up 2 minutes datahub-datahub-actions-1 dc1ab1b5ba11 linkedin/datahub-frontend-react:v0.9.0 "/bin/sh -c ./start.…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:9002->9002/tcp, :::9002->9002/tcp datahub-frontend-react 0f3cdc10702a confluentinc/cp-kafka:5.4.0 "/etc/confluent/dock…" 6 minutes ago Exited (1) 54 seconds ago broker 1bd32618a1dc linkedin/datahub-gms:v0.9.0 "/bin/sh -c /datahub…" 6 minutes ago Up 2 minutes (health: starting) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp datahub-gms 6d15a3d61287 linkedin/datahub-elasticsearch-setup:v0.9.0 "/bin/sh -c 'if [ \"$…" 6 minutes ago Exited (0) About a minute ago elasticsearch-setup 44d937e52af2 acryldata/datahub-mysql-setup:v0.9.0 "/bin/sh -c 'dockeri…" 6 minutes ago Exited (1) About a minute ago mysql-setup 98903cc4dfc1 confluentinc/cp-zookeeper:5.4.0 "/etc/confluent/dock…" 6 minutes ago Exited (139) 55 seconds ago zookeeper 75db48a15e61 elasticsearch:7.9.3 "/tini -- /usr/local…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp elasticsearch db14ec5fb272 mysql:5.7 "docker-entrypoint.s…" 6 minutes ago Up 6 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql
i
GMS is failing because kafka and schema registry are failing, which in turn is because zookeeper fails which reports the following:
Copy code
zookeeper                  | # A fatal error has been detected by the Java Runtime Environment:
zookeeper                  | #
zookeeper                  | #  SIGSEGV (0xb) at pc=0x00007fdb504a5532, pid=1, tid=0x00007fdb51067700
zookeeper                  | #
zookeeper                  | # JRE version: OpenJDK Runtime Environment (Zulu 8.38.0.13-CA-linux64) (8.0_212-b04) (build 1.8.0_212-b04)
zookeeper                  | # Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
zookeeper                  | # Problematic frame:
zookeeper                  | # C  [libc.so.6+0x36532]  abort+0x232
zookeeper                  | #
zookeeper                  | # Core dump written. Default location: //core or core.1
zookeeper                  | #
zookeeper                  | # An error report file with more information is saved as:
zookeeper                  | # //hs_err_pid1.log
zookeeper                  | #
zookeeper                  | # If you would like to submit a bug report, please visit:
zookeeper                  | #   <http://www.azulsystems.com/support/>
zookeeper                  | # The crash happened outside the Java Virtual Machine in native code.
zookeeper                  | # See problematic frame for where to report the bug.
zookeeper                  | #
How powerful is the machine where you are deploying DataHub?
We need at least: 2 CPUs, 8GB RAM, 2GB Swap area, and 10GB disk space for docker alone. See: https://datahubproject.io/docs/get-started-with-datahub#prerequisites
I would even say, recommended would be 4 CPUs, 10GB RAM, 3GB swap and 30GB disk
c
My machine has 8 CPUs, 32 GB RAM, 14.3GB swap and 52.5GB disk
Ok maybe I don't have enough RAM for Docker ? I can see a total of 32 GB RAM but only 6.7 GB MemFree
i
You can run 
docker stats
to check for runtime pods
If you’re looking for configs take a look at: https://stackoverflow.com/a/44533437
c
Thks. But I think i'm good with the required config
i
I find it very strange that MySQL is asking for 16GB of memory
Did you just run 
datahub docker quickstart
?
c
Yes
Sorry not exactly, as requested, i've added one option :
Copy code
datahub docker quickstart --version v0.9.0
i
I see, will try to reproduce and get back to you
I can not reproduce this on my side:
Copy code
(datahub-env) ➜  ~ datahub version
DataHub CLI version: 0.9.0.4
Python version: 3.8.9 (default, Apr 13 2022, 08:48:06)
[Clang 13.1.6 (clang-1316.0.21.2.5)]
(datahub-env) ➜  ~ docker ps -a
CONTAINER ID   IMAGE                           COMMAND       CREATED      STATUS       PORTS     NAMES
b6143ac7f0f6   moby/buildkit:buildx-stable-1   "buildkitd"   2 days ago   Up 3 hours             buildx_buildkit_naughty_gates0
(datahub-env) ➜  ~ datahub docker quickstart --version v0.9.0
Detected M1 machine
No Datahub Neo4j volume found, starting with elasticsearch as graph service.
To use neo4j as a graph backend, run
`datahub docker quickstart --quickstart-compose-file ./docker/quickstart/docker-compose.quickstart.yml`
from the root of the datahub repo

Fetching docker-compose file <https://raw.githubusercontent.com/datahub-project/datahub/master/docker/quickstart/docker-compose-without-neo4j-m1.quickstart.yml> from GitHub
Pulling docker images...
Finished pulling docker images!

[+] Running 11/11
 ⠿ Container elasticsearch              Started                                                                                                                                                                  0.8s
 ⠿ Container mysql                      Started                                                                                                                                                                  0.9s
 ⠿ Container zookeeper                  Started                                                                                                                                                                  0.8s
 ⠿ Container broker                     Started                                                                                                                                                                  1.3s
 ⠿ Container mysql-setup                Started                                                                                                                                                                  1.4s
 ⠿ Container datahub-gms                Started                                                                                                                                                                  1.4s
 ⠿ Container elasticsearch-setup        Started                                                                                                                                                                  1.3s
 ⠿ Container schema-registry            Started                                                                                                                                                                  1.6s
 ⠿ Container datahub-datahub-actions-1  Started                                                                                                                                                                  1.9s
 ⠿ Container datahub-frontend-react     Started                                                                                                                                                                  2.0s
 ⠿ Container kafka-setup                Started                                                                                                                                                                  2.1s
.............
[+] Running 11/11
 ⠿ Container zookeeper                  Running                                                                                                                                                                  0.0s
 ⠿ Container mysql                      Running                                                                                                                                                                  0.0s
 ⠿ Container broker                     Running                                                                                                                                                                  0.0s
 ⠿ Container elasticsearch              Running                                                                                                                                                                  0.0s
 ⠿ Container mysql-setup                Started                                                                                                                                                                  0.6s
 ⠿ Container elasticsearch-setup        Started                                                                                                                                                                  0.5s
 ⠿ Container schema-registry            Running                                                                                                                                                                  0.0s
 ⠿ Container datahub-gms                Running                                                                                                                                                                  0.0s
 ⠿ Container kafka-setup                Started                                                                                                                                                                  0.6s
 ⠿ Container datahub-datahub-actions-1  Running                                                                                                                                                                  0.0s
 ⠿ Container datahub-frontend-react     Running                                                                                                                                                                  0.0s
.............
[+] Running 11/11
 ⠿ Container elasticsearch              Running                                                                                                                                                                  0.0s
 ⠿ Container mysql                      Running                                                                                                                                                                  0.0s
 ⠿ Container zookeeper                  Running                                                                                                                                                                  0.0s
 ⠿ Container elasticsearch-setup        Started                                                                                                                                                                  0.7s
 ⠿ Container mysql-setup                Started                                                                                                                                                                  0.7s
 ⠿ Container datahub-gms                Running                                                                                                                                                                  0.0s
 ⠿ Container broker                     Running                                                                                                                                                                  0.0s
 ⠿ Container datahub-datahub-actions-1  Running                                                                                                                                                                  0.0s
 ⠿ Container datahub-frontend-react     Running                                                                                                                                                                  0.0s
 ⠿ Container schema-registry            Running                                                                                                                                                                  0.0s
 ⠿ Container kafka-setup                Started                                                                                                                                                                  0.6s
......
✔ DataHub is now running
Ingest some demo data using `datahub docker ingest-sample-data`,
or head to <http://localhost:9002> (username: datahub, password: datahub) to play around with the frontend.
Need support? Get in touch on Slack: <https://slack.datahubproject.io/>
(datahub-env) ➜  ~ docker ps -a
CONTAINER ID   IMAGE                                         COMMAND                   CREATED              STATUS                        PORTS                                        NAMES
3971993fa207   linkedin/datahub-kafka-setup:v0.9.0           "/bin/sh -c ./kafka-…"    About a minute ago   Exited (0) 34 seconds ago                                                  kafka-setup
2c4e5bf43fb6   linkedin/datahub-frontend-react:v0.9.0        "/bin/sh -c ./start.…"    About a minute ago   Up About a minute (healthy)   0.0.0.0:9002->9002/tcp                       datahub-frontend-react
0cba3550c11d   acryldata/datahub-actions:head                "/bin/sh -c 'dockeri…"    About a minute ago   Up About a minute                                                          datahub-datahub-actions-1
b4665ecbb1a3   confluentinc/cp-schema-registry:7.2.0         "/etc/confluent/dock…"    About a minute ago   Up About a minute             0.0.0.0:8081->8081/tcp                       schema-registry
7a9e2467a77e   linkedin/datahub-gms:v0.9.0                   "/bin/sh -c /datahub…"    About a minute ago   Up About a minute (healthy)   0.0.0.0:8080->8080/tcp                       datahub-gms
8aace50e7415   confluentinc/cp-kafka:7.2.0                   "/etc/confluent/dock…"    About a minute ago   Up About a minute             0.0.0.0:9092->9092/tcp                       broker
0351bb6e7cb4   linkedin/datahub-elasticsearch-setup:v0.9.0   "/bin/sh -c 'if [ \"$…"   About a minute ago   Exited (0) 47 seconds ago                                                  elasticsearch-setup
3d4d56f4a038   acryldata/datahub-mysql-setup:v0.9.0          "/bin/sh -c 'dockeri…"    About a minute ago   Exited (0) 46 seconds ago                                                  mysql-setup
2ddf58003df5   mariadb:10.5.8                                "docker-entrypoint.s…"    About a minute ago   Up About a minute             0.0.0.0:3306->3306/tcp                       mysql
3dbaff4698da   confluentinc/cp-zookeeper:7.2.0               "/etc/confluent/dock…"    About a minute ago   Up About a minute             2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zookeeper
8a015ebfa5a2   elasticsearch:7.9.3                           "/tini -- /usr/local…"    About a minute ago   Up About a minute (healthy)   0.0.0.0:9200->9200/tcp, 9300/tcp             elasticsearch
b6143ac7f0f6   moby/buildkit:buildx-stable-1                 "buildkitd"               2 days ago           Up 3 hours                                                                 buildx_buildkit_naughty_gates0
(datahub-env) ➜  ~
What is the output of 
docker version
? I have:
Copy code
Client:
 Cloud integration: v1.0.25
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Thu May 12 09:20:34 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.9.1 (81317)
 Engine:
  Version:          20.10.16
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.10
  Git commit:       f756502
  Built:            Thu May 12 09:14:19 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.4
  GitCommit:        212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc:
  Version:          1.1.1
  GitCommit:        v1.1.1-0-g52de29d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
c
I have this one : Client: Docker Engine - Community Version: 20.10.21 API version: 1.41 Go version: go1.18.7 Git commit: baeda1f Built: Tue Oct 25 180216 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.21 API version: 1.41 (minimum version 1.12) Go version: go1.18.7 Git commit: 3056208 Built: Tue Oct 25 180001 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.9 GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f runc: Version: 1.1.4 GitCommit: v1.1.4-0-g5fd4c4d docker-init: Version: 0.19.0 GitCommit: de40ad0
i
This is very strange… Could you try the same steps on a different OS?
c
I've already succeed to install it on my Mac and found it really insteresting. The reason why I've tried to install it after on RHEL.
i
DataHub is now working on Mac? Is this no longer an issue?
In RHEL it still does not work, is that it?
c
Yes, I still can't start it (unable to quickstart)
I've found other issues where MySQL is asking for 16GB of memory. I've tried to fix it adding ulimits to MySQL service in the docker-compose.yml file but it doesn't work. It seems that the docker-compose.yml is overwritten during the quickstart. Do you know how to add ulimits to MySQL container ? ulimits: nproc: 65535 nofile: soft: 20000 hard: 40000
i
Hello Stefan, You can use the docker-compose that the 
datahub
cli downloads, found under 
~/.datahub/quickstart/
and run docker-compose from it directly once you modify the MySQL container with the limits you want.
See the accepted answer in SO
7 Views
Open in Slack
PreviousNext
Powered by