https://datahubproject.io logo
Join Slack
Powered by
Hi, I'm new to DataHub and trying to set it up loc...
# troubleshoot
j
Hi, I'm new to DataHub and trying to set it up locally on my Mac. I'm using Colima as a docker desktop replacement. I get the following error when I run: 
datahub docker quickstart
or 
sudo datahub docker quickstart
Copy code
> sudo datahub docker quickstart

Docker doesn't seem to be running. Did you start it?

~ .........................................................................................................................................................................
> datahub docker quickstart
Docker doesn't seem to be running. Did you start it?
Can someone help me figure this out? My 
docker
setup is running.
b
i dont think it can run if you have to use sudo https://docs.docker.com/engine/install/linux-postinstall/
j
Neither of them worked. Tried with and without sudo. I’m using Colima instead of docker desktop since docker desktop now requires a paid version.
Docker is running - I checked my other containers.
b
missing 
compose
plugin?
j
Docker compose command works for other yaml files though
b
not familiar with colima. well, if you want to eliminate possibilities, then try downloading one of the yaml in https://github.com/datahub-project/datahub/tree/master/docker/quickstart and just docker compose up the yaml file, since the CLI basically does the same thing.
j
Thanks, will try that out.
Running the docker compose manually kinda worked. Thanks for the tip. But the containers are not starting up. I only see these logs:
Copy code
elasticsearch-setup           | 2023/02/07 17:29:23 Problem with request: Get "<http://elasticsearch:9200>": dial tcp: lookup elasticsearch on 127.0.0.11:53: no such host. Sleeping 1s
quickstart-datahub-actions-1  | 2023/02/07 17:29:23 Problem with request: Get "<http://datahub-gms:8080/health>": dial tcp 172.20.0.10:8080: connect: connection refused. Sleeping 1s
datahub-gms                   | 2023/02/07 17:29:23 Problem with request: Get "<http://neo4j:7474>": dial tcp: lookup neo4j on 127.0.0.11:53: no such host. Sleeping 1s
datahub-gms                   | 2023/02/07 17:29:23 Problem with request: Get "<http://elasticsearch:9200>": dial tcp: lookup elasticsearch on 127.0.0.11:53: no such host. Sleeping 1s
elasticsearch-setup           | 2023/02/07 17:29:24 Problem with request: Get "<http://elasticsearch:9200>": dial tcp: lookup elasticsearch on 127.0.0.11:53: no such host. Sleeping 1s
quickstart-datahub-actions-1  | 2023/02/07 17:29:24 Problem with request: Get "<http://datahub-gms:8080/health>": dial tcp 172.20.0.10:8080: connect: connection refused. Sleeping 1s
datahub-upgrade               | 2023-02-07 17:29:24.632  INFO 1 --- [           main] org.reflections.Reflections              : Reflections took 307 ms to scan 8 urls, producing 34 keys and 72 values
datahub-gms                   | 2023/02/07 17:29:25 Problem with request: Get "<http://elasticsearch:9200>": dial tcp: lookup elasticsearch on 127.0.0.11:53: no such host. Sleeping 1s
datahub-gms                   | 2023/02/07 17:29:25 Problem with request: Get "<http://neo4j:7474>": dial tcp: lookup neo4j on 127.0.0.11:53: no such host. Sleeping 1s
broker                        | [2023-02-07 17:29:25,222] INFO [LogLoader partition=_schemas-0, dir=/var/lib/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.UnifiedLog$)
elasticsearch-setup           | 2023/02/07 17:29:25 Problem with request: Get "<http://elasticsearch:9200>": dial tcp: lookup elasticsearch on 127.0.0.11:53: no such host. Sleeping 1s
For schema-registry:
Copy code
schema-registry               | [2023-02-07 17:29:53,735] INFO [AdminClient clientId=adminclient-2] Disconnecting from node 1 due to socket connection setup timeout. The timeout value is 10639 ms. (org.apache.kafka.clients.NetworkClient)
schema-registry               | [2023-02-07 17:29:53,910] WARN [AdminClient clientId=adminclient-2] Error connecting to node broker:29092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
schema-registry               | java.net.UnknownHostException: broker: Name or service not known
schema-registry               | 	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
schema-registry               | 	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
schema-registry               | 	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
schema-registry               | 	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
schema-registry               | 	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
schema-registry               | 	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1378)
schema-registry               | 	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
schema-registry               | 	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
schema-registry               | 	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:125)
schema-registry               | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:528)
schema-registry               | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:485)
schema-registry               | 	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
schema-registry               | 	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1103)
schema-registry               | 	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:363)
schema-registry               | 	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1202)
schema-registry               | 	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1462)
schema-registry               | 	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1405)
schema-registry               | 	at java.base/java.lang.Thread.run(Thread.java:829)
schema-registry               | 	at org.apache.kafka.common.utils.KafkaThread.run(KafkaThread.java:64)
b
There are quite a few pieces to running datahub in docker, this includes various prerequisites such as kafka, mysql, and elasticsearch. The first log shows that the elasticsearch components are not running in your docker environment and the second is referring to the kafka broker. The quickstart command helps to coordinate the various datahub components and their prerequisite storage systems. Additionally depending on the type of mac (m1) it may also require slightly different images, another layer that is handled by the quickstart scripts. Looking at the quickstart script it appears to be looking for the docker socket. I am curious as to where this socket might be in your environment and then perhaps we can expand the checks there to use it.
j
That file is available at that path. Also, I am running the 
docker-compose.quickstart.yml
- it should be able to handle all pre-requisites needed for DataHub, right? I see that it starts Kafka, and other related services including zookeeper and schema-registry. But, after sometime, the broker and schema-regsitry service just exit/shutdown. Not sure why. This happens when I docker-compose the confluent kafka image as well.
b
Can you share some of the logs from zookeeper and the broker. Let’s start there, those should be independent of schema-registry. The quickstart process actually by default doesn’t use the the neo4j version, but this one
If on an m1, it also (or perhaps instead of) pulls in this one
j
Mine is a Intel Mac Below are some log snippets from schema registry, broker and zookeeper:
Copy code
schema-registry               | [2023-02-08 22:43:02,952] ERROR Error while getting broker list. (io.confluent.admin.utils.ClusterStatus)
schema-registry               | java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
schema-registry               | 	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
schema-registry               | 	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
schema-registry               | 	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
schema-registry               | 	at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:147)
schema-registry               | 	at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:149)
Copy code
broker                        | [2023-02-08 22:43:06,084] WARN Client session timed out, have not heard from server in 40004ms for session id 0x0 (org.apache.zookeeper.ClientCnxn)
broker                        | [2023-02-08 22:43:06,086] WARN An exception was thrown while closing send thread for session 0x0. (org.apache.zookeeper.ClientCnxn)
broker                        | org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 40004ms for session id 0x0
broker                        | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258)
Copy code
quickstart-datahub-actions-1  | 2023/02/08 22:44:29 Problem with request: Get "<http://datahub-gms:8080/health>": dial tcp 172.20.0.10:8080: connect: connection refused. Sleeping 1s
datahub-frontend-react        | 2023-02-08 22:44:29,888 [kafka-producer-network-thread | datahub-frontend] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=datahub-frontend] Error connecting to node broker:29092 (id: -1 rack: null)
datahub-frontend-react        | java.net.UnknownHostException: broker
datahub-frontend-react        | 	at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
datahub-frontend-react        | 	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1524)
datahub-frontend-react        | 	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1382)
datahub-frontend-react        | 	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
datahub-frontend-react        | 	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:331)
datahub-frontend-react        | 	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238)
datahub-frontend-react        | 	at java.base/java.lang.Thread.run(Thread.java:829)
datahub-gms                   | 2023/02/08 22:44:29 Problem with dial: dial tcp: lookup broker on 127.0.0.11:53: no such host. Sleeping 1s
datahub-gms                   | 2023/02/08 22:44:30 Problem with request: Get "<http://elasticsearch:9200>": dial tcp: lookup elasticsearch on 127.0.0.11:53: no such host. Sleeping 1s
quickstart-datahub-actions-1  | 2023/02/08 22:44:30 Problem with request: Get "<http://datahub-gms:8080/health>": dial tcp 172.20.0.10:8080: connect: connection refused. Sleeping 1s
datahub-gms                   | 2023/02/08 22:44:30 Problem with dial: dial tcp: lookup broker on 127.0.0.11:53: no such host. Sleeping 1s
datahub-frontend-react        | 2023-02-08 22:44:30,977 [kafka-producer-network-thread | datahub-frontend] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=datahub-frontend] Error connecting to node broker:29092 (id: -1 rack: null)
datahub-frontend-react        | java.net.UnknownHostException: broker
datahub-frontend-react        | 	at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
datahub-frontend-react        | 	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1524)
datahub-frontend-react        | 	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1382)
datahub-frontend-react        | 	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
datahub-frontend-react        | 	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
datahub-frontend-react        | 	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
datahub-frontend-react        | 	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:331)
datahub-frontend-react        | 	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238)
datahub-frontend-react        | 	at java.base/java.lang.Thread.run(Thread.java:829)
Is it possible that this is a memory or cpu issue on my Mac? I do see that there is still about 10GB of memory free.
b
This looks more like a networking issue, they are all disconnected from each other. There is a bit at the top of the compose file which should be putting all the components on the same virtual network, are you by chance overriding this or setting any other network related options?
j
@brainy-tent-14503 I haven't changed any defaults. Here is what I have in the yaml:
Copy code
networks:
  default:
    name: datahub_network
@brainy-tent-14503 I was able to get datahub running locally using helm charts.
Copy code
> k get po
NAME                                                READY   STATUS      RESTARTS        AGE
prerequisites-zookeeper-0                           1/1     Running     1 (3h54m ago)   3h56m
prerequisites-cp-schema-registry-5f89dd4974-5bngr   2/2     Running     3 (3h53m ago)   3h56m
prerequisites-mysql-0                               1/1     Running     1 (3h54m ago)   3h56m
prerequisites-kafka-0                               1/1     Running     3 (3h53m ago)   3h56m
elasticsearch-master-0                              1/1     Running     1 (3h54m ago)   3h56m
datahub-datahub-frontend-6c6d8cc9c5-rbd2c           1/1     Running     0               3h46m
datahub-acryl-datahub-actions-f7b9c4977-c7wrw       1/1     Running     0               3h46m
datahub-elasticsearch-setup-job-r7ssx               0/1     Completed   0               17m
datahub-kafka-setup-job-h4j9h                       0/1     Completed   0               17m
datahub-mysql-setup-job-fph7m                       0/1     Completed   0               15m
datahub-datahub-system-update-job-mw9db             0/1     Completed   0               14m
datahub-datahub-gms-6d5d66bb58-fctzw                1/1     Running     0               12m
datahub-nocode-migration-job-rdnx9                  0/1     Completed   0               12m
But when I try to call the schema registry API:
Copy code
> curl <http://prerequisites-cp-schema-registry:8081/schemas>
curl: (6) Could not resolve host: prerequisites-cp-schema-registry
I tried to update the value in 
values.yaml
and give it a try - it still gives me the same error. Any idea why this happens?
b
That should be correct. There should be a k8 service with that name. 
kubectl get svc
would expect to see a service called 
prerequisites-cp-schema-registry
listening on port 8081. Where are you seeing this error from? Looks like kafka setup ran successfully. This all looks good.
j
Yes, I do see it in the list of services:
Copy code
prerequisites-cp-schema-registry   ClusterIP      10.43.241.119   <none>         8081/TCP,5556/TCP               4h19m
b
Ok, and where are you connecting from? Are you port-forwarding the serrvice?
In order to port-forward, something like 
kubectl -n <namespace> port-forward svc/prerequisites-cp-schema-registry 8081:8081
would expose the port from that service on localhost. Most folks want to point to the frontend and point a browser to it.
j
Yeah - I was expecting the schema registry URL to be accessible outside of the k8s cluster namespace. Port-forwarding works. Thanks!
37 Views
Open in Slack
PreviousNext
Powered by