Getting Below Error while Creating a New Ingestion...
# troubleshoot
j
Getting Below Error while Creating a New Ingestion Source
Copy code
~~~~ Execution Summary ~~~~

RUN_INGEST - {'errors': [],
 'exec_id': '04fe03c0-c639-4f9f-aa52-6d488a12738f',
 'infos': ['2022-09-20 09:01:36.448196 [exec_id=04fe03c0-c639-4f9f-aa52-6d488a12738f] INFO: Starting execution for task with name=RUN_INGEST',
           '2022-09-20 09:01:40.487800 [exec_id=04fe03c0-c639-4f9f-aa52-6d488a12738f] INFO: stdout=Elapsed seconds = 0\n'
           '  --report-to TEXT                Provide an destination to send a structured\n'
           'This version of datahub supports report-to functionality\n'
           'datahub  ingest run -c /tmp/datahub/ingest/04fe03c0-c639-4f9f-aa52-6d488a12738f/recipe.yml --report-to '
           '/tmp/datahub/ingest/04fe03c0-c639-4f9f-aa52-6d488a12738f/ingestion_report.json\n'
           '[2022-09-20 09:01:39,455] INFO     {datahub.cli.ingest_cli:179} - DataHub CLI version: 0.8.44\n'
           '[2022-09-20 09:01:40,097] ERROR    {datahub.entrypoints:192} - \n'
           'Traceback (most recent call last):\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/entrypoints.py", line 149, in main\n'
           '    sys.exit(datahub(standalone_mode=False, **kwargs))\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/click/core.py", line 1130, in __call__\n'
           '    return self.main(*args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/click/core.py", line 1055, in main\n'
           '    rv = self.invoke(ctx)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/click/core.py", line 1657, in invoke\n'
           '    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/click/core.py", line 1657, in invoke\n'
           '    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/click/core.py", line 1404, in invoke\n'
           '    return ctx.invoke(self.callback, **ctx.params)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/click/core.py", line 760, in invoke\n'
           '    return __callback(*args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func\n'
           '    return f(get_current_context(), *args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/telemetry/telemetry.py", line 347, in wrapper\n'
           '    raise e\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/telemetry/telemetry.py", line 299, in wrapper\n'
           '    res = func(*args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/utilities/memory_leak_detector.py", line 102, in '
           'wrapper\n'
           '    return func(*args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 182, in run\n'
           '    pipeline_config = load_config_file(\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/configuration/config_loader.py", line 79, in '
           'load_config_file\n'
           '    config = resolve_env_variables(raw_config)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/configuration/config_loader.py", line 43, in '
           'resolve_env_variables\n'
           '    new_dict[k] = resolve_env_variables(v)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/configuration/config_loader.py", line 43, in '
           'resolve_env_variables\n'
           '    new_dict[k] = resolve_env_variables(v)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/configuration/config_loader.py", line 47, in '
           'resolve_env_variables\n'
           '    new_dict[k] = resolve_element(v)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/datahub/configuration/config_loader.py", line 15, in '
           'resolve_element\n'
           '    return expandvars(element, nounset=True)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/expandvars.py", line 475, in expandvars\n'
           '    return expand(vars_, nounset=nounset)\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/expandvars.py", line 434, in expand\n'
           '    return "".join(buff) + expand_var(\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/expandvars.py", line 171, in expand_var\n'
           '    return expand_modifier_var(\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/expandvars.py", line 216, in expand_modifier_var\n'
           '    return expand_advanced(\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/expandvars.py", line 294, in expand_advanced\n'
           '    return expand_offset(\n'
           '  File "/tmp/datahub/ingest/venv-postgres-0.8.44/lib/python3.9/site-packages/expandvars.py", line 344, in expand_offset\n'
           '    raise OperandExpected(var, offset_str)\n'
           "expandvars.OperandExpected: CMDB: operand expected (error token is 'Credential')\n"
           '[2022-09-20 09:01:40,098] ERROR    {datahub.entrypoints:195} - Command failed: \n'
           "\tCMDB: operand expected (error token is 'Credential').\n"
           '\tRun with --debug to get full stacktrace.\n'
           "\te.g. 'datahub --debug ingest run -c /tmp/datahub/ingest/04fe03c0-c639-4f9f-aa52-6d488a12738f/recipe.yml --report-to "
           "/tmp/datahub/ingest/04fe03c0-c639-4f9f-aa52-6d488a12738f/ingestion_report.json'\n",
           "2022-09-20 09:01:40.488106 [exec_id=04fe03c0-c639-4f9f-aa52-6d488a12738f] INFO: Failed to execute 'datahub ingest'",
           '2022-09-20 09:01:40.488406 [exec_id=04fe03c0-c639-4f9f-aa52-6d488a12738f] INFO: Caught exception EXECUTING '
           'task_id=04fe03c0-c639-4f9f-aa52-6d488a12738f, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n'
           '    self.event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 89, in run_until_complete\n'
           '    return f.result()\n'
           '  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
           '    raise self._exception\n'
           '  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
           '    result = coro.send(None)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 142, in execute\n'
           '    raise TaskError("Failed to execute \'datahub ingest\'")\n'
           "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"]}
Execution finished with errors.
d
Please, can you elaborate it a bit more what did you try to do?
j
Yaml Format
Copy code
source:
    type: postgres
    config:
        include_tables: true
        database: cdb
        password: '${Credential}'
        profiling:
            enabled: false
        host_port: 'cdb:5432'
        include_views: true
        stateful_ingestion:
            enabled: true
        username: cdb_user
pipeline_name: 'urn:li:dataHubIngestionSource:-a0cb641a0f18'
@dazzling-judge-80093 I am trying to add a New Ingestion Source - Postgres and getting above error while running it
d
It seems the problem is with the Credential
error token is 'Credential'
even though the error message is not very user friendly
j
The credentials are correct. I have verified them a couple of times. Can it be due to VPN configuration settings ? And yeah, I agree. Error message can be a bit user friendly.