https://datahubproject.io logo
Join Slack
Powered by
Good morning all! I am trying to run the `./docker...
# troubleshoot
m
Good morning all! I am trying to run the 
./docker/dev.sh
script to get started on local development. When I do this I run into the following issue:
Copy code
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.dev.yml pull
[+] Running 9/12
 ⠿ zookeeper Pulled                                                                                                                                                                                            1.2s
 ⠿ broker Pulled                                                                                                                                                                                               1.2s
 ⠿ elasticsearch-setup Warning                                                                                                                                                                                 1.5s
 ⠿ kafka-setup Warning                                                                                                                                                                                         1.7s
 ⠿ mysql Pulled                                                                                                                                                                                                1.2s
 ⠿ datahub-actions Pulled                                                                                                                                                                                      1.3s
 ⠿ mysql-setup Pulled                                                                                                                                                                                          1.4s
 ⠿ datahub-frontend-react Warning                                                                                                                                                                              1.5s
 ⠿ neo4j Pulled                                                                                                                                                                                                1.3s
 ⠿ datahub-gms Pulled                                                                                                                                                                                          1.4s
 ⠿ schema-registry Pulled                                                                                                                                                                                      1.4s
 ⠿ elasticsearch Pulled                                                                                                                                                                                        1.2s
WARNING: Some service image(s) must be built from source by running:
    docker compose build datahub-frontend-react elasticsearch-setup kafka-setup
3 errors occurred:
	* Error response from daemon: manifest for linkedin/datahub-frontend-react:debug not found: manifest unknown: manifest unknown
	* Error response from daemon: manifest for linkedin/datahub-elasticsearch-setup:debug not found: manifest unknown: manifest unknown
	* Error response from daemon: manifest for linkedin/datahub-kafka-setup:debug not found: manifest unknown: manifest unknown
running 
COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 DOCKER_DEFAULT_PLATFORM="$(uname -m)" docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.dev.yml build kafka-setup
exits with exit code 0 so that works fine. My local build also succeeds with no issues. I've had a look through the documentation and this channel but I have not found an issue similar to this so I am at a bit of a loss what to do here.
b
Hey @many-solstice-66904, the dev.sh tries to use the images tagged with 
:debug
https://github.com/datahub-project/datahub/blob/master/docker/docker-compose.dev.yml#L15 This needs to be built with 
--build
as here: https://github.com/datahub-project/datahub/blob/master/docker/dev.sh#L36
can you the build happened?
m
running 
COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 DOCKER_DEFAULT_PLATFORM="$(uname -m)" docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.dev.yml up --build
instead of 
./dev.sh
does start up the docker-compose! Thanks!
Though the original issue still remains, as I can't just run 
./dev.sh
and running the command in my previous message
When I run just the last command in 
./dev.sh
the compose starts up but I encounter various errors, mostly along the lines of connection refused:
Copy code
broker                     | [2023-01-30 15:12:19,394] INFO Socket connection established, initiating session, client: /172.22.0.6:49622, server: zookeeper/172.22.0.3:2181 (org.apache.zookeeper.ClientCnxn)
broker                     | [2023-01-30 15:12:19,410] INFO Session establishment complete on server zookeeper/172.22.0.3:2181, sessionid = 0x1000013d5f60004, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
broker                     | [2023-01-30 15:12:19,417] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
kafka-setup                | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.22.0.6:29092) could not be established. Broker may not be available.
broker                     | [2023-01-30 15:12:19,853] INFO Cluster ID = yIs4V-aYRlGx6hUBVaiwLw (kafka.server.KafkaServer)
broker                     | [2023-01-30 15:12:19,865] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
broker                     | kafka.common.InconsistentClusterIdException: The Cluster ID yIs4V-aYRlGx6hUBVaiwLw doesn't match stored clusterId Some(tdMvimAORMKeYVuotsChWw) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
For anyone running into the first Issue I had in this thread, running the 
./docker/dev.sh
script with docker-compose 
v1.29.2
fixed it. (previously used docker compose v2.*)
b
Thanks @many-solstice-66904 reporting this back 🙌
s
Hi, I'm getting the same issue, side question, how did you downgrade the docker composer version? docker composer 1 seems to be deprecated. The documentation says:
This install scenario is no longer supported.
m
Docker-compose 1 is indeed deprecated, I was lucky since my distro's package manager still had it in an archive so I could install it manually (see here). The code itself can be downloaded here https://github.com/docker/compose/releases/tag/1.29.2 . For the install instruction you can look at https://github.com/archlinux/svntogit-community/blob/1581ba2d368ee9833f83c31b9b479cde4943a963/trunk/PKGBUILD and perform them manually
6 Views
Open in Slack
PreviousNext
Powered by