Hey all, I’m running into the following issue in UI-triggered runs of a docker-setup. The error seems pretty self-explanatory tbh but I can’t seem to resolve it —

"[2022-09-27 15:41:58,839] ERROR    {datahub.ingestion.run.pipeline:127} - 'Did not find a registered class for bigquery-beta'\n"
           '[2022-09-27 15:41:58,840] INFO     {datahub.cli.ingest_cli:119} - Starting metadata ingestion\n'
           '[2022-09-27 15:41:58,845] INFO     {datahub.cli.ingest_cli:137} - Finished metadata ingestion\n'
           "[2022-09-27 15:41:59,052] ERROR    {datahub.entrypoints:188} - Command failed with 'Pipeline' object has no attribute 'source'. Run with "
           '--debug to get full trace\n'
           '[2022-09-27 15:41:59,053] INFO     {datahub.entrypoints:191} - DataHub CLI version: 0.8.42 at '
           '/tmp/datahub/ingest/venv-bigquery-beta-0.8.42/lib/python3.10/site-packages/datahub/__init__.py\n',
           "2022-09-27 15:41:59.221819 [exec_id=a037e7da-0b12-4056-aba4-d83b2915f3cb] INFO: Failed to execute 'datahub ingest'",
           '2022-09-27 15:41:59.222128 [exec_id=a037e7da-0b12-4056-aba4-d83b2915f3cb] INFO: Caught exception EXECUTING '
           'task_id=a037e7da-0b12-4056-aba4-d83b2915f3cb, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 123, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 203, in execute\n'
           '    raise TaskError("Failed to execute \'datahub ingest\'")\n'
           "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"]}
Execution finished with errors.

Seems that datahub CLI version is pinned to 0.8.42 somehow, but I installed my CLI with version specification

>= 0.8.45

and can confirm this in my command line. I’ve run

datahub docker quickstart

a few times, even purging images/containers and I’m still seeing this. What am I missing?

Hey @blue-crowd-84759 yes this is inconvenient. The way to work around this is to provide your desired cli version in the advanced section on the last page of the ingestion form.

Thank you that did it! Would be worth having it in the how to upgrade datahub docs

@blue-crowd-84759 the doco is here https://datahubproject.io/docs/ui-ingestion/#advanced-running-with-a-specific-cli-version

Gotcha, but without at least a backlink to that, it still leaves this section incomplete https://datahubproject.io/docs/quickstart/#upgrading-your-local-datahub