https://datahubproject.io logo
Join Slack
Powered by
Hi I am doing Postgres Ingestion to the data in th...
# ingestion
s
Hi I am doing Postgres Ingestion to the data in the Postgtres Container. My 
config file
looks like this. 
source:
type: "postgres"
config:
# Coordinates
 
host_port: "localhost:5435"
database: "srcdb"
# Credentials
 
username: "source"
password: "gsRABSy6xvWsSTE3"
sink:
type: "datahub-rest"
config:
server: "<http://localhost:8080>"
I am getting the following error:
h
Hi @steep-airplane-60304 this looks like connection issue. Is postgres database running on your localhost on port 5432 ?
s
Yes. I am running it in a docker container
h
have you mapped the ports to host machine for postgres container ?
Copy code
ports:
    - 5432:5432
s
Yes. The mapping is 
ports:
-5435:5432
h
are you running the recipe on host machine or inside container ?
s
Host Machine
h
Also could you share entire error log here ? it'll helpful
s
Sure
(datahub-env) ~/t/datahub ❯❯❯ datahub ingest -c recipe.dhub.yaml
[2022-10-03 17:01:15,873] INFO     {datahub.cli.ingest_cli:182} - DataHub CLI version: 0.8.45.1
[2022-10-03 17:01:29,222] ERROR    {datahub.entrypoints:192} -
Traceback (most recent call last):
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f1667949430>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connectionpool.py", line 815, in urlopen
return self.urlopen(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connectionpool.py", line 815, in urlopen
return self.urlopen(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connectionpool.py", line 815, in urlopen
return self.urlopen(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /config (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1667949430>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/ingestion/api/common.py", line 67, in __init__
self.graph = DataHubGraph(datahub_api) if datahub_api is not None else None
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/ingestion/graph/client.py", line 73, in __init__
self.test_connection()
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/emitter/rest_emitter.py", line 157, in test_connection
response = self._session.get(f"{self._gms_server}/config")
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /config (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1667949430>: Failed to establish a new connection: [Errno 111] Connection refused'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 151, in __init__
self.ctx = PipelineContext(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/ingestion/api/common.py", line 69, in __init__
raise Exception("Failed to connect to DataHub") from e
Exception: Failed to connect to DataHub
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 197, in run
pipeline = Pipeline.create(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 317, in create
return cls(
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 160, in __init__
self._record_initialization_failure(e, "Failed to set up framework context")
File "/home/essayyzed/temp/datahub/datahub-env/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 129, in _record_initialization_failure
raise PipelineInitError(msg) from e
datahub.ingestion.run.pipeline.PipelineInitError: Failed to set up framework context
[2022-10-03 17:01:29,223] ERROR    {datahub.entrypoints:195} - Command failed:
Failed to set up framework context due to
'Failed to connect to DataHub' due to
'HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /config (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1667949430>: Failed to establish a new connection: [Errno 111] Connection refused'))'.
Run with --debug to get full stacktrace.
e.g. 'datahub --debug ingest -c recipe.dhub.yaml'
h
have you started datahub locally ? using `datahub docker quickstart`or helm ?
s
I setup it using the 
pip
h
You'll have to deploy a datahub instance as shown here, in order to use datahub sink (after pip install) https://datahubproject.io/docs/quickstart/
s
Okay
I just wanted to use the CLI version only. How i can acheive that
h
s
It is necessary to install and use dataHub by using the above method if one is only concern with CLI version only
3 Views
Open in Slack
PreviousNext
Powered by