Hello
how can i deploy datahub with Docker compose...
# all-things-deploymentf
fierce-agent-11572
06/07/2023, 9:00 AMHello
how can i deploy datahub with Docker compose ( not with datahub quickstart command please)
when i use the cmd :
Copy code
datahub docker quickstart --quickstart-compose-file quickstart/docker-compose.quickstart.yml✅ 1
b
better-orange-49102
06/07/2023, 10:10 AMdocker compose -p datahub -f quickstart/docker-compose.quickstart.ymldocker-composedocker composef
fierce-agent-11572
06/07/2023, 3:29 PMhello
my version
Docker Compose version v2.18.1
fierce-agent-11572
06/07/2023, 3:38 PMwhen i execute the command i have the help without error message
i must add up in the last of the command but the command start all components exept front and gms
Copy code
Usage: docker compose [OPTIONS] COMMAND
Define and run multi-container applications with Docker.b
better-orange-49102
06/07/2023, 11:08 PMIf you don't want to start certain containers just comment out the container in the yml. If you can't start up containers just look at the logs if the failed containers.
f
fierce-agent-11572
06/08/2023, 7:54 AMi don't have error in docker compose but i don't have front and gms container started
fierce-agent-11572
06/08/2023, 8:00 AMthe frontend and gms stay only created
[ssm-user@ip-10-32-248-253 quickstart]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
cea600e89990 acryldata/datahub-actions:head "/bin/sh -c 'dockeri…" 5 minutes ago Created
datahub-actions
0a93d31b01ba linkedin/datahub-frontend-react:head "/bin/sh -c ./start.…" 5 minutes ago Created
datahub-frontend-react
9617d3c9d269 linkedin/datahub-gms:head "/bin/sh -c /datahub…" 5 minutes ago Created
datahub-gms
a1b87d210d53 acryldata/datahub-upgrade:head "java -jar /datahub/…" 5 minutes ago Created
datahub-upgrade
e85d4f4f8607 linkedin/datahub-kafka-setup:head "/bin/sh -c ./kafka-…" 5 minutes ago Created
kafka-setup
f5738dd138e5 confluentinc/cp-schema-registry:7.2.2 "/etc/confluent/dock…" 5 minutes ago Created
schema-registry
b145616925f4 confluentinc/cp-kafka:7.2.2 "/etc/confluent/dock…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp
broker
b0e6eee74471 acryldata/datahub-mysql-setup:head "/bin/sh -c 'dockeri…" 5 minutes ago Created
mysql-setup
3fe1d7ebac60 linkedin/datahub-elasticsearch-setup:head "/bin/sh -c 'if [ \"$…" 5 minutes ago Exited (0) 4 minutes ago elasticsearch-setup
c2f44cf3d0cf elasticsearch:7.10.1 "/tini -- /usr/local…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp elasticsearch
3198c1253ba3 mysql:5.7 "docker-entrypoint.s…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql
8eee2ce7a34b confluentinc/cp-zookeeper:7.2.2 "/etc/confluent/dock…" 5 minutes ago Up 5 minutes (healthy) 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp zookeeper
108af434a3dd neo4j:4.4.9-community "tini -g -- /startup…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:76
b
better-orange-49102
06/08/2023, 8:17 AMdocker logs gmsf
fierce-agent-11572
06/08/2023, 9:04 AMi don't have an error but the front and gms container are not started
b
better-orange-49102
06/08/2023, 9:06 AMi have no idea how to diagnose if
docker logs datahub-gmsf
fierce-agent-11572
06/08/2023, 9:19 AMno logs it's stay like this
Copy code
[root@ip-10-32-248-117 docker]# datahub docker quickstart --quickstart-compose-file quickstart/docker-compose.quickstart.yml
[2023-06-08 11:13:04,441] INFO {datahub.cli.quickstart_versioning:144} - Saved quickstart config to /root/.datahub/quickstart/quickstart_version_mapping.yaml.
[2023-06-08 11:13:04,441] INFO {datahub.cli.docker_cli:643} - Using quickstart plan: composefile_git_ref='master' docker_tag='head'
Pulling docker images...
This may take a while depending on your network bandwidth.
[+] Pulling 18/18
✔ datahub-gms Pulled 0.8s
✔ datahub-upgrade Pulled 0.8s
✔ elasticsearch-setup Pulled 0.8s
✔ mysql-setup Pulled 0.8s
✔ datahub-frontend-react Pulled 0.8s
✔ datahub-actions Pulled 0.8s
✔ kafka-setup Pulled 0.8s
✔ elasticsearch Pulled 0.8s
✔ mysql Pulled 0.8s
✔ neo4j 5 layers [⣿⣿⣿⣿⣿] 0B/0B Pulled 6.6s
✔ bd159e379b3b Pull complete 1.8s
✔ 0411b6cb0e94 Pull complete 4.3s
✔ 5e77b499dc6a Pull complete 4.3s
✔ a9b56f145a78 Pull complete 4.4s
✔ 1f0c2624d51b Pull complete 5.3s
✔ zookeeper Pulled 0.8s
✔ broker Pulled 0.8s
✔ schema-registry Pulled 0.8s
Finished pulling docker images!
Starting up DataHub...
[+] Building 0.0s (0/0)
[+] Running 19/19
✔ Network datahub_network Created 0.0s
✔ Volume "datahub_mysqldata" Created 0.0s
✔ Volume "datahub_neo4jdata" Created 0.0s
✔ Volume "datahub_esdata" Created 0.0s
✔ Volume "datahub_broker" Created 0.0s
✔ Volume "datahub_zkdata" Created 0.0s
✘ Container mysql Error 12.0s
✔ Container zookeeper Healthy 14.0s
✔ Container neo4j Started 2.3s
✔ Container elasticsearch Healthy 4.5s
✔ Container mysql-setup Created 0.0s
✔ Container elasticsearch-setup Started 4.3s
✘ Container broker Error 27.2s
✔ Container schema-registry Created 0.0s
✔ Container kafka-setup Created 0.0s
✔ Container datahub-upgrade Created 0.0s
✔ Container datahub-gms Created 0.0s
✔ Container datahub-frontend-react Created 0.0s
✔ Container datahub-actions Createdfierce-agent-11572
06/08/2023, 9:20 AMand when i do docker ps from othen console this is the result
Copy code
sudo docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESee8e09778249 confluentinc/cp-kafka:7.2.2 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp
broker
6baeb7c256a8 confluentinc/cp-zookeeper:7.2.2 "/etc/confluent/dock…" 6 minutes ago Up 6 minutes (healthy) 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp
zookeeper
2282727c0d94 mysql:5.7 "docker-entrypoint.s…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql
cbd8521a79cb elasticsearch:7.10.1 "/tini -- /usr/local…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp elasticsearch
54888c7e43fb neo4j:4.4.9-community "tini -g -- /startup…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp neo4j
[ssm-user@ip-10-32-248-117 bin]$b
better-orange-49102
06/08/2023, 9:22 AMmy guess is that you need to diagnose the broker and mysql failed startups, i think docker compose is ordering the container startups and GMS needs mysql and broker to start up successfully.
b
best-rose-86507
06/09/2023, 7:00 AMI'm getting the same issue, did anyone manage to solve it ?
✔ Network datahub_network Created✔ Volume "datahub_mysqldata" Created✔ Volume "datahub_esdata" Created✔ Volume "datahub_zkdata" Created✔ Volume "datahub_broker" Created✔ Container elasticsearch Healthy✔ Container zookeeper Healthy✘ Container mysql Error✔ Container elasticsearch-setup Started✔ Container mysql-setup Created✘ Container broker Error✔ Container schema-registry Created✔ Container kafka-setup Created✔ Container datahub-upgrade Created✔ Container datahub-gms Created✔ Container datahub-actions Created✔ Container datahub-frontend-react Createdh
hallowed-kilobyte-916
06/10/2023, 6:29 AMdatahub docker quickstart --quickstart-compose-file <path to compose file>f
fierce-agent-11572
06/12/2023, 8:32 AMsame issue with this command also 😞
fierce-agent-11572
06/21/2023, 3:00 PMwhich docker compose file can be used in production env ?