https://datahubproject.io logo
Join Slack
Powered by
```pi@raspberrypi:~/workspace/datahub $ ./docker/q...
# all-things-deployment
i
Copy code
pi@raspberrypi:~/workspace/datahub $ ./docker/quickstart.sh
Pulling zookeeper       ... done
Pulling neo4j         ... pulling from library/neo4j
Pulling mysql         ... pulling from library/mysql
Pulling broker        ... done
Pulling schema-registry    ... done
Pulling kafka-setup      ... done
Pulling elasticsearch     ... pulling from library/elasticsearch
Pulling kibana        ... pulling from library/kibana
Pulling elasticsearch-setup  ... done
Pulling datahub-gms      ... done
Pulling datahub-mce-consumer ... done
Pulling datahub-frontend-react ... done
Pulling datahub-mae-consumer ... done
Pulling kafka-rest-proxy   ... done
Pulling kafka-topics-ui    ... done
Pulling schema-registry-ui  ... done

ERROR: for elasticsearch no matching manifest for unknown in the manifest list entries

ERROR: for mysql no matching manifest for unknown in the manifest list entries

ERROR: for neo4j no matching manifest for unknown in the manifest list entries


ERROR: for kibana no matching manifest for unknown in the manifest list entries
ERROR: no matching manifest for unknown in the manifest list entries
g
i think this has to do with rasberry pi's support for these images
reading here, it seems like there are alternative images needed for rpi support, for example: https://hub.docker.com/r/hypriot/rpi-mysql/
i
Awesome, thanks! So I should manually pull down these images? Is there anything I need to do to get things pointed to these new images?
g
i would start by trying to get this images set up ad-hoc using docker commands
to productionalize, i would create parallel docker-compose files for the rpi-mysql, rpi-elasticsearch, etc
i
As in, I shouldn't use the quickstart.sh file?
g
correct, the quickstart.sh file is pointing to the non-rpi images
you can also edit the quickstart file locally if youd like
b
Would recommend simply replacing the images for the affected images in 
docker-compose.yml
For those compatible with rpi
i
Awesome I'll give that a try.
Hmm got them all resolved except neo4j
g
hmm- sounds like you'll need to do some more research to see if there's a better neo4j image out there
alternatively, it might be easier to run this on an instance in aws, google cloud etc
i
I'll take your advice and go with a Linux VM.
m
@icy-easter-2378: if you had trouble running it on mac, please post the errors here
i
Absolutely will do.
Forgive the wordy error that's about to follow.
Copy code
(base) blars-book-pro:datahub bryanlarson$ ./docker/quickstart.sh
Traceback (most recent call last):
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.28.6', 'console_scripts', 'docker-compose')())
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/compose/cli/main.py", line 198, in perform_command
    project = project_from_options('.', options)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/Cellar/docker-compose/1.28.6/libexec/lib/python3.9/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
m
@icy-easter-2378: make sure Docker Desktop is running on the mac?
i
Haha yep that did it. I told you I was a dummy when it comes to docker.
There are some strange exceptions in the logs. For example:
Copy code
Caused by: com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://datahub-gms:8080/charts>
datahub-frontend-react    | 	at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
datahub-frontend-react    | 	at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponseImpl(ResponseFutureImpl.java:130)
datahub-frontend-react    | 	at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponse(ResponseFutureImpl.java:94)
datahub-frontend-react    | 	at com.linkedin.chart.client.Charts.search(Charts.java:108)
datahub-frontend-react    | 	at com.linkedin.datahub.graphql.types.chart.ChartType.search(ChartType.java:97)
datahub-frontend-react    | 	at com.linkedin.datahub.graphql.resolvers.search.SearchResolver.lambda$get$1(SearchResolver.java:53)
datahub-frontend-react    | 	... 6 common frames omitted
Not sure why it's trying to reach out to another server. My understanding was that all required services were running locally.
To be clear, the site does appear to be running successfully, other than some suspicious log entries.
Thanks @green-football-43791, @big-carpet-38439, and @mammoth-bear-12532 for your help and patience.
e
Hey. Can you run 
COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml up --build -d elasticsearch-setup
this should solve the above issue
i
Run that, then run the quickstart.sh script?
e
if everything is already running
things should just work. Is it still showing the error?
the above command should be run from docker directory! sry I forgot to add this info
i
Yeah I cd'd into that directory
I stopped the server to run the command. I should have just waited for your response.
e
ah no worries
i
Lot of exceptions in the logs
Here's one:
Copy code
java.net.UnknownHostException: broker: Name or service not known
schema-registry           | 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
schema-registry           | 	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
schema-registry           | 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
schema-registry           | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
schema-registry           | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
schema-registry           | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
schema-registry           | 	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
schema-registry           | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
schema-registry           | 	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
schema-registry           | 	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
schema-registry           | 	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
schema-registry           | 	at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:69)
schema-registry           | 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1126)
schema-registry           | 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1017)
schema-registry           | 	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:538)
schema-registry           | 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
schema-registry           | 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
schema-registry           | 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
schema-registry           | 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
schema-registry           | 	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:484)
schema-registry           | 	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1267)
schema-registry           | 	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
schema-registry           | 	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1168)
schema-registry           | 	at io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread.doWork(KafkaStoreReaderThread.java:152)
schema-registry           | 	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
e
are all the docker containers up and running? can you try 
docker container ls
?
i
No, several are not. Also this exception is pretty early on:
Copy code
org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
datahub-mce-consumer      | 	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185)
datahub-mce-consumer      | 	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53)
datahub-mce-consumer      | 	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360)
datahub-mce-consumer      | 	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158)
datahub-mce-consumer      | 	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122)
datahub-mce-consumer      | 	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894)
datahub-mce-consumer      | 	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:163)
datahub-mce-consumer      | 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
datahub-mce-consumer      | 	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142)
datahub-mce-consumer      | 	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
datahub-mce-consumer      | 	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
datahub-mce-consumer      | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
datahub-mce-consumer      | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
datahub-mce-consumer      | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
datahub-mce-consumer      | 	at com.linkedin.metadata.kafka.MceConsumerApplication.main(MceConsumerApplication.java:12)
datahub-mce-consumer      | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
datahub-mce-consumer      | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
datahub-mce-consumer      | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
datahub-mce-consumer      | 	at java.lang.reflect.Method.invoke(Method.java:498)
datahub-mce-consumer      | 	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
datahub-mce-consumer      | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
datahub-mce-consumer      | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
datahub-mce-consumer      | 	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
datahub-mce-consumer      | Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
datahub-mce-consumer      | 	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:825)
datahub-mce-consumer      | 	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:631)
datahub-mce-consumer      | 	at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:164)
datahub-mce-consumer      | 	at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createConsumerWithAdjustedProperties(DefaultKafkaConsumerFactory.java:160)
datahub-mce-consumer      | 	at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:135)
datahub-mce-consumer      | 	at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createConsumer(DefaultKafkaConsumerFactory.java:109)
datahub-mce-consumer      | 	at org.springframework.kafka.listener.AbstractMessageListenerContainer.checkTopics(AbstractMessageListenerContainer.java:305)
datahub-mce-consumer      | 	at org.springframework.kafka.listener.ConcurrentMessageListenerContainer.doStart(ConcurrentMessageListenerC
Whoops that got cut off. I'll edit that.
e
no worries. Can you try running docker/nuke.sh before running quickstart?
Seems like we got into a bad state stopping the previous set of containers
i
Absolutely.
I've consistently seen this type of thing in the logs:
Copy code
datahub-mae-consumer      | 2021/04/06 19:59:38 Problem with dial: dial tcp 172.19.0.8:29092: connect: connection refused. Sleeping 1s
datahub-mae-consumer      | 2021/04/06 19:59:38 Problem with request: Get <http://neo4j:7474>: dial tcp 172.19.0.3:7474: connect: connection refused. Sleeping 1s
elasticsearch-setup       | 2021/04/06 19:59:39 Problem with request: Get <http://elasticsearch:9200>: dial tcp 172.19.0.2:9200: getsockopt: connection refused. Sleeping 1s
datahub-gms               | 2021/04/06 19:59:39 Problem with dial: dial tcp 172.19.0.8:29092: connect: connection refused. Sleeping 1s
datahub-mae-consumer      | 2021/04/06 19:59:39 Problem with request: Get <http://elasticsearch:9200>: dial tcp 172.19.0.2:9200: connect: connection refused. Sleeping 1s
datahub-gms               | 2021/04/06 19:59:39 Problem with request: Get <http://neo4j:7474>: dial tcp 172.19.0.3:7474: connect: connection refused. Sleeping 1s
datahub-gms               | 2021/04/06 19:59:40 Problem with request: Get <http://elasticsearch:9200>: dial tcp 172.19.0.2:9200: connect: connection refused. Sleeping 1s
kafka-rest-proxy          | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.19.0.8:29092) could not be established. Broker may not be available.
kafka-setup               | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.19.0.8:29092) could not be established. Broker may not be available.
neo4j, elasticsearch, and kafka. I assumed it was a lazy initialization and eventually they would catch up.
e
yeah that is not an issue. it’s just waiting for dependencies to come up before starting the container
i
Hmm I'm seeing those kafka stack traces again. Every couple seconds.
Copy code
20:02:51.724 [mce-consumer-job-client-0-C-1] WARN  o.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-mce-consumer-job-client-2, groupId=mce-consumer-job-client] Error connecting to node broker:29092 (id: 1 rack: null)
datahub-mce-consumer      | java.net.UnknownHostException: broker
datahub-mce-consumer      | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
datahub-mce-consumer      | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
datahub-mce-consumer      | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
datahub-mce-consumer      | 	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
And docker is reporting about half of the services have exited.
m
@icy-easter-2378: was this resolved?
e
Closing up the thread. The issue was caused by lack of RAM on the docker instance. Kafka brokers died of OOM. Once more memory was given, all the pods started up!
✅ 1
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by