i modified the sample_recipe.yml in docker/ingesti...
# ingestionb
better-orange-49102
04/11/2021, 3:58 PMi modified the sample_recipe.yml in docker/ingestion/quickstart.sh to point to a postgresql docker instance to try it out. when the sink is set to console the output looks ok, but when i set the sink to datahub-rest and tried to start the ingestion container, it threw the following error, how might i troubleshoot this?
Copy code
ingestion | [2021-04-11 15:51:17,986] ERROR {datahub.ingestion.run.pipeline:41} - failed to write record with workunit public.mytable with ('Unable to emit metadata to DataHub GMS', {'message': "HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /datasets?action=ingest (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f14a3261790>: Failed to establish a new connection: [Errno 111] Connection refused'))"}) and info {'message': "HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /datasets?action=ingest (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f14a3261790>: Failed to establish a new connection: [Errno 111] Connection refused'))"}better-orange-49102
04/11/2021, 4:03 PMto give more context, i started datahub using the quickstart.sh, but didnt want to load the sample data and instead tried to point it to a postgres instance that is also connected to the datahub_network.
sample recipe looks like this:
Copy code
---
source:
type: postgres
config:
username: postgres
password: postgres
database: mydb
host_port: postgresql-dev:5432
sink:
type: "datahub-rest"
config:
server: '<http://localhost:8080>'g
gray-shoe-75895
04/11/2021, 5:47 PMlooks like it was unable to connect to the backend - can you run
datahub check local-dockerb
better-orange-49102
04/13/2021, 3:41 PMhi Harshal, sorry it took a while to test.
so i amended the docker-compose in docker/ingestion/docker-compose.yml to make sure the container stays alive, then opened an interactive terminal in the container.
Copy code
datahub check local-docker Copy code
The following issues were detected:
- Docker doesn't seem to be running. Did you start it?better-orange-49102
04/13/2021, 3:42 PMI'm not very good with docker, but it seems like i needed to do some port binding for the ingestion container?
better-orange-49102
04/13/2021, 3:54 PMoh, changing the server in the sample_recipe from https://localhost:8080 to https://datahub-gms:8080 works for me
👍 1
g
gray-shoe-75895
04/13/2021, 5:05 PMAh glad you figured it out!
gray-shoe-75895
04/13/2021, 5:05 PMI didn't realize you were running
datahub