https://datahubproject.io logo
Join Slack
Powered by
Hey, im having issues running the `datahub docker ...
# troubleshoot
p
Hey, im having issues running the 
datahub docker quickstart
, a few of the containers keep restarting and exiting... I dont see and obvious issues, no erros in mysql-setup, gms has error with:
Copy code
2023-02-22 17:21:43 2023/02/22 16:21:43 Problem with dial: dial tcp 172.18.0.8:29092: connect: connection refused. Sleeping 1s
2023-02-22 17:21:44 2023/02/22 16:21:44 Timeout after 4m0s waiting on dependencies to become available: [<http://elasticsearch:9200> <tcp://mysql:3306> <tcp://broker:29092>]
Zoopkeer looks like is not running:
Copy code
2023-02-22 17:18:48 [2023-02-22 16:18:48,129] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
2023-02-22 17:18:48 [2023-02-22 16:18:48,369] WARN Close of session 0x0 (org.apache.zookeeper.server.NIOServerCnxn)
2023-02-22 17:18:48 java.io.IOException: ZooKeeperServer not running
2023-02-22 17:18:48     at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:544)
2023-02-22 17:18:48     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:332)
2023-02-22 17:18:48     at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
2023-02-22 17:18:48     at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
2023-02-22 17:18:48     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2023-02-22 17:18:48     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2023-02-22 17:18:48     at java.base/java.lang.Thread.run(Thread.java:829)
2023-02-22 17:18:48 [2023-02-22 16:18:48,434] WARN Unexpected exception (org.apache.zookeeper.server.WorkerService)
2023-02-22 17:18:48 java.lang.NullPointerException
I dont really know what to do with this...
✅ 1
a
Hi it looks like you are maybe having some connection issues or running into zookeeper problems
Starting off, could you try setting the version? If you’re on the most recent update this looks like 
datahub docker quickstart --version=v0.10.0
if that doesn’t work, you might want to try a reset with 
datahub docker nuke
(this will completely wipe all data, so backup if you have anything in quickstart) and then run the above versioned command again
if you’ve tried that and it still isn’t working, let me know here
Be sure to follow the prerequesites with installing/updating docker/docker compose/ python/ and the CLI here https://datahubproject.io/docs/quickstart
p
I tried what you suggested, but still same issue
I created a new environment and run the command, also i am running on mac so i run: 
datahub docker quickstart --arch m1
a
You shouldn’t need the arch command, it will detect automatically!
p
yeah, same results even after nuke
g
@purple-oil-61897 It something with docker networking, GMS and ZK container interaction are broken try 
docker network inspect datahub_network
to see all containers in datahub_network it should looking something like this
Copy code
[
    {
        "Name": "datahub_network",
        "Id": "5dfe142b76a3f3a6a144b427e6afa05e54e529aab47cea0908152db8c83e7eee",
        "Created": "2022-09-22T10:15:41.144322507Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.22.0.0/16",
                    "Gateway": "172.22.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "01893edb765e5412824c1fabf2e29d0ad69d834853bd10c6e40f7ffad82c30b4": {
                "Name": "datahub-upgrade",
                "EndpointID": "75ad1b0771e83f2a896493147b9bba542b093196dd56134563c15ea2370e4636",
                "MacAddress": "02:42:ac:16:00:02",
                "IPv4Address": "172.22.0.2/16",
                "IPv6Address": ""
            },
            "01c6211710e389e58830671d10ca7c00c8b029474a62b39e8cb5b444cb537893": {
                "Name": "datahub-datahub-actions-1",
                "EndpointID": "0a843bd7d1ef4290df385a68bf104bc29c9ab5bdcf214acd6e77063db6a401a3",
                "MacAddress": "02:42:ac:16:00:0e",
                "IPv4Address": "172.22.0.14/16",
                "IPv6Address": ""
            },
            "166349a82166b913211651984f0ad762e17198f7f950faebad0be67cd71ea25f": {
                "Name": "zookeeper",
                "EndpointID": "02cd7c1b95a23480f082b9f4ed124d34f08149107a79076a5ca8b0a5fd7ecbac",
                "MacAddress": "02:42:ac:16:00:03",
                "IPv4Address": "172.22.0.3/16",
                "IPv6Address": ""
            },
            "29fd17cb215c56a185ccff50d548720b84e54092661779671b9bce2f8ca8b13d": {
                "Name": "elasticsearch",
                "EndpointID": "24fdf8a25e2c8804d39eab8ebd44b8bc819b2e86c7494242423871af499578a6",
                "MacAddress": "02:42:ac:16:00:04",
                "IPv4Address": "172.22.0.4/16",
                "IPv6Address": ""
            },
            "34ee8447d88b8878a751376d003c4faeadfc1d9254eb163182636bf1e0102218": {
                "Name": "mysql",
                "EndpointID": "6f6eb8cffc1dee51c25782016dc40031463eb4b81d3cbd6b6a290c8539ec02de",
                "MacAddress": "02:42:ac:16:00:05",
                "IPv4Address": "172.22.0.5/16",
                "IPv6Address": ""
            },
            "57496d1b31803af0e38a65c03873ade2cdbf26db09d2ac47b73186047a35ff31": {
                "Name": "broker",
                "EndpointID": "bc4e7173af5d35936b60d984a7eae9da4ac292c879725a2b6794db41af3fd813",
                "MacAddress": "02:42:ac:16:00:07",
                "IPv4Address": "172.22.0.7/16",
                "IPv6Address": ""
            },
            "7672664ed2b8125ee53965faefd32bb31d92614070af3d5c15bc15f503bf26a9": {
                "Name": "schema-registry",
                "EndpointID": "cbb1126e1ed8ac291b9f87399f7e4ffc9cf5f521730e1f6dc6fdccebb9874749",
                "MacAddress": "02:42:ac:16:00:0c",
                "IPv4Address": "172.22.0.12/16",
                "IPv6Address": ""
            },
            "77d7e19d93bfd66309f47ed5a46f9f9d9dcf5b1af28e944f5dd8bf1132309006": {
                "Name": "airflow_dev_ram_image-airflow-webserver-1",
                "EndpointID": "905c5bddb6922d2f52fb13474888aed1bfaf2307fb9bed2d776ffce9bf5b4ab3",
                "MacAddress": "02:42:ac:16:00:09",
                "IPv4Address": "172.22.0.9/16",
                "IPv6Address": ""
            },
            "85fd496047f4f23fab8817d44160406287ea934ce6902fcfc7762a7cc7693e75": {
                "Name": "datahub-frontend-react",
                "EndpointID": "6e4042d71ba6846548e8ff052325d5d232f1f5e9d796f0d55b6f7e0170770e59",
                "MacAddress": "02:42:ac:16:00:0f",
                "IPv4Address": "172.22.0.15/16",
                "IPv6Address": ""
            },
            "912de90c1b49335c99f7f029be2076c6fff5cd14bae8f24f4dace0dac8b07274": {
                "Name": "airflow_dev_ram_image-airflow-scheduler-1",
                "EndpointID": "0f094c68a72b22a78650b11eaee7397dc33655075ea91354649d42ab8f8a4640",
                "MacAddress": "02:42:ac:16:00:08",
                "IPv4Address": "172.22.0.8/16",
                "IPv6Address": ""
            },
            "ce01ebd8533615443fe172329e2fc16115c7082db55c62967eadbf3f4a18788d": {
                "Name": "airflow_dev_ram_image-postgres-1",
                "EndpointID": "85e4dd68b7c1441e3b25ab8fdcd63fd8cae925f029dc74e01617eadf7ac5112a",
                "MacAddress": "02:42:ac:16:00:06",
                "IPv4Address": "172.22.0.6/16",
                "IPv6Address": ""
            },
            "e18ea25dcdfc9dc5bfc26b54daab1dab1b0284cf8d56806349299a004382f3fc": {
                "Name": "datahub-gms",
                "EndpointID": "9e76797a618246d0821a40e21e6d187d370e88de2be570f2a28f96da358c8445",
                "MacAddress": "02:42:ac:16:00:0b",
                "IPv4Address": "172.22.0.11/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "datahub",
            "com.docker.compose.version": "2.10.2"
        }
    }
]
You may try stop all container, prune docker networks
Copy code
docker network prune
p
i run your command, after network prune i get this:
Copy code
[
    {
        "Name": "datahub_network",
        "Id": "1fdefbb7892b99bad465aec22f3b2b0ae0e92abc30a33b83e9308144bfb43ea2",
        "Created": "2023-02-23T14:21:01.263749132Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.22.0.0/16",
                    "Gateway": "172.22.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "017b622466088b895cc991e03b98a2517568d7d78bc85ce87f0d549218452b0f": {
                "Name": "mysql",
                "EndpointID": "e89d9a654e7e08f4614d1a53129a6767528e8b5ef8acbad11eecb49d570105f3",
                "MacAddress": "02:42:ac:16:00:05",
                "IPv4Address": "172.22.0.5/16",
                "IPv6Address": ""
            },
            "1c85d06eef7e0985ef9d95b10fb54c28997b3fccfbeea90b70646dc1f9751c21": {
                "Name": "datahub-gms",
                "EndpointID": "5078a7f2f18ec0f38e57d591b2e2b4d56cff86d0f3155beb679c1598b6f3d29f",
                "MacAddress": "02:42:ac:16:00:09",
                "IPv4Address": "172.22.0.9/16",
                "IPv6Address": ""
            },
            "2d85dccfb974af5ef8123519db6823f423b348f80e3c46ef13215f39249a7373": {
                "Name": "elasticsearch",
                "EndpointID": "9baad458d6b0c6585a09b09a3bb40ede5f4704cc479612b9266ef673260ec0da",
                "MacAddress": "02:42:ac:16:00:03",
                "IPv4Address": "172.22.0.3/16",
                "IPv6Address": ""
            },
            "6fc38094f2bb3ea3c06296875d871942eb29b390901c24ea2753b04fa896c044": {
                "Name": "datahub-frontend-react",
                "EndpointID": "e8b5d6d35eeeedc72bd8664a0b7b6d79dee3eb5ea03f5bdfbdef301342b6b7ea",
                "MacAddress": "02:42:ac:16:00:0c",
                "IPv4Address": "172.22.0.12/16",
                "IPv6Address": ""
            },
            "896402ae7287dfb89c699538ca9ee4dd637f0d00ec76515b2f8193a91ad13591": {
                "Name": "broker",
                "EndpointID": "f3fa3d99cdd787dd355b792fbe039b9543c5e70ae9ee0a96438b7674f0d3458b",
                "MacAddress": "02:42:ac:16:00:08",
                "IPv4Address": "172.22.0.8/16",
                "IPv6Address": ""
            },
            "9a897ac2ec2eaddf760c7793f6ca32694d41e775912a16026346abe8835e8be4": {
                "Name": "datahub-upgrade",
                "EndpointID": "ba5a9fe1cae5835921080a3ec491ee172e08c08995e83b78e727b4a1fd8e9c5a",
                "MacAddress": "02:42:ac:16:00:02",
                "IPv4Address": "172.22.0.2/16",
                "IPv6Address": ""
            },
            "b49ce2cf84a308ce3860ed753488efadc886a77ca83b2e5878728d9cf0459d5a": {
                "Name": "zookeeper",
                "EndpointID": "47a765fe507f4f2baf0634b59a277e6e2626b37bce9e938b203323a2b3dde2c4",
                "MacAddress": "02:42:ac:16:00:04",
                "IPv4Address": "172.22.0.4/16",
                "IPv6Address": ""
            },
            "d03341f3017b0fe1d4996587cab7c146a6f032620f45090ef5088fe578e3e05c": {
                "Name": "datahub-datahub-actions-1",
                "EndpointID": "262fb6404c58e606fb6d538560fddf8b97632970ea7daa25fd38dd3abd55d494",
                "MacAddress": "02:42:ac:16:00:0b",
                "IPv4Address": "172.22.0.11/16",
                "IPv6Address": ""
            },
            "df6647333044b7c4212c2d73826ce94c1cf6cb7ca0edcd40c22d4bb1fc9e2c2d": {
                "Name": "schema-registry",
                "EndpointID": "5292f0f04ced13fcf950deed62a85435b5bb85859435489698e99b4d89e71662",
                "MacAddress": "02:42:ac:16:00:0a",
                "IPv4Address": "172.22.0.10/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "datahub",
            "com.docker.compose.version": "2.15.1"
        }
    }
]
g
this is looks good, is the issue still persist?
p
yes
ok i didnt another rerun, at it looks like its running now
thank you for your help!
g
I’m glad it worked !
p
sth is still not working properly, the UI is extremely slow and is not showing any of the items i added
this is what i get:
image.png
mysql:
Copy code
2023-02-23 17:07:06 2023-02-23T16:07:06.545708Z 31 [Note] Got an error reading communication packets
a
CC: @orange-night-91387, @purple-oil-61897, this could be to ES reindexing- has it sped up at all since you posted?
o
How much memory/resources are you giving to Docker?
p
no ist still not working...
image.png
could someone jump on a quick call with me to check this out please
o
Sure I can take a quick look 🙂
Sent you a Slack huddle invite, if you need it to be through a different app let me know
Hopping off the huddle, guessing you're not available at the moment. As far as what else could be going wrong here can you please provide the logs for GMS & Frontend? This seems to be an issue with GMS not starting up (seeing the blank UI like that indicates it doesn't get any data back from GMS)
p
ok sorry, i didnt see that, im avaiable now
o
It seems like somehow you've gotten the amd64 image for ElasticSearch and you need the arm one for an M1 Mac. If you run: 
docker images | grep elastic
docker inspect --format='{{index .RepoDigests 0}}' <IMAGE ID>
-> with IMAGE ID from the first command what do you get?
p
Copy code
elasticsearch@sha256:7cd88158f6ac75d43b447fdd98c4eb69483fa7bf1be5616a85fe556262dc864a
@orange-night-91387 What does this tell us?
o
It lets me verify the exact image you have pulled to match it up with mine. Looks like we have the same hash, but mine is working so the issue lies with either Docker Desktop or differences in our machine archs 😕 What version of Docker Desktop are you using and double checking you are using an M1 Pro Mac on Monterey 12.x?
p
I have M1 Pro Mac on Ventura 13.2x, Docker Desktop v4.16.2
Before I har an older version of docker which, and i updated it to try and fix this issue
a
is there any way we could run this on a VM?
p
You mean like ecs? First i want to run it locally before involving our devops
a
You could even just set it up on a local VM
to get around the kernel issues
same set up, no deployment, gets around whatever is going on with your architecture
p
yeah i dont think i wanna spend more time setting up a vm for this, thanks for the suggestion tho
15 Views
Open in Slack
PreviousNext
Powered by