https://datahubproject.io logo
Join Slack
Powered by
Hi, I'm trying to setup ingestion from the cli, I ...
# troubleshoot
r
Hi, I'm trying to setup ingestion from the cli, I very simple yaml file im using but i keep getting 
PipelineInitError
any help would be appreciated!
✅ 1
Copy code
source:
  type: redshift
  config:
    # Coordinates
    host_port: '${host}:${port}'
    database: '${dbname}'
    username: '${username}'
    password: '${password}'

sink:
  type: datahub-rest
  config:
    server: MYSERVER
    token: MYTOKEN
and the error is
Copy code
[2023-01-26 17:22:43,813] ERROR    {datahub.entrypoints:226} - Command failed: Failed to set up framework context: Failed to instantiate a valid DataHub Graph instance
Traceback (most recent call last):
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/api/common.py", line 63, in __init__
    self.graph = DataHubGraph(datahub_api) if datahub_api is not None else None
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/graph/client.py", line 72, in __init__
    self.test_connection()
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/emitter/rest_emitter.py", line 148, in test_connection
    config: dict = response.json()
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 114, in _add_init_error_context
    yield
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 162, in __init__
    self.ctx = PipelineContext(
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/api/common.py", line 67, in __init__
    raise Exception(
Exception: Failed to instantiate a valid DataHub Graph instance

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/entrypoints.py", line 184, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 350, in wrapper
    raise e
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 302, in wrapper
    res = func(*args, **kwargs)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/utilities/memory_leak_detector.py", line 95, in wrapper
    return func(ctx, *args, **kwargs)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 179, in run
    pipeline = Pipeline.create(
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 303, in create
    return cls(
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 161, in __init__
    with _add_init_error_context("set up framework context"):
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/john.corrales/projects/datahub/ingestion/venv/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 116, in _add_init_error_context
    raise PipelineInitError(f"Failed to {step}: {e}") from e
datahub.ingestion.run.pipeline.PipelineInitError: Failed to set up framework context: Failed to instantiate a valid DataHub Graph instance
make: *** [dry_run] Error 1
b
from cli, a "sink" is required. either console, file, datahub-rest etc
r
yeah i have that in the file too, just didnt copy it. I updated the snippet
b
what is the MYSERVER looking like? are you connecting directly to gms (8080) or? the error looks like it cant find a working connection
r
i have it deployed as part of a fargate cluster, i have the gms container exposed through the load balancer. I'm pretty sure its making a connection, when i put an invalid address, it hangs for a bit and i eventually get a 
Failed to establish a new connection
error.
the 
JSONDecodeError
i'm getting is making me think maybe the response from the gms container is an error or something instead of the expected json response.
b
im not sure how to debug from here unfortunately. Its part of a kubernetes setup or something? maybe can port-forward to gms and try a localhost:8080 connection?
r
No worries, thanks for your help! Was just wondering if there was a quick solution i was missing. Ive been hacking on the request, and found its returning html (looks like from the ui container) so maybe my lb config is broken.
Circling back here, i was able to connect but using the gms proxy in the frontend 
<<mydatahubserver>>/api/gms
Still don't have it completely working, but got past that last issue. thanks again for the help!
Open in Slack
PreviousNext
Powered by