Hi all, Just noticed that an ingestion started failing post upgrade to the latest v0.9.0 (previous version was v0.8.44). Specifically a scheduled ingestion for Looker, configured via the UI. The next scheduled execution post upgrade produced an error as shown in the log below. No change to the source configuration itself had been made, for reference:

source:
    type: looker
    config:
        extract_owners: true
        base_url: '<our url>'
        skip_personal_folders: true
        include_deleted: false
        client_secret: '${LOOKER_CLIENT_SECRET}'
        extract_usage_history: true
        env: prod
        platform: snowflake
        client_id: '${LOOKER_CLIENT_ID}'
pipeline_name: 'urn:li:dataHubIngestionSource:<uuid here>'

For now I have explicitly set the CLI version on the source to 0.8.45 as a workaround - it then runs without issue. The ERROR log entry as follows:

'[2022-10-15 05:03:37,869] ERROR    {datahub.entrypoints:192} - \n'
           'Traceback (most recent call last):\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/entrypoints.py", line 149, in main\n'
           '    sys.exit(datahub(standalone_mode=False, **kwargs))\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/click/core.py", line 1130, in __call__\n'
           '    return self.main(*args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/click/core.py", line 1055, in main\n'
           '    rv = self.invoke(ctx)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/click/core.py", line 1657, in invoke\n'
           '    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/click/core.py", line 1657, in invoke\n'
           '    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/click/core.py", line 1404, in invoke\n'
           '    return ctx.invoke(self.callback, **ctx.params)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/click/core.py", line 760, in invoke\n'
           '    return __callback(*args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func\n'
           '    return f(get_current_context(), *args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 347, in wrapper\n'
           '    raise e\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 299, in wrapper\n'
           '    res = func(*args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/utilities/memory_leak_detector.py", line 95, in '
           'wrapper\n'
           '    return func(ctx, *args, **kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 212, in run\n'
           '    loop.run_until_complete(run_func_check_upgrade(pipeline))\n'
           '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 166, in '
           'run_func_check_upgrade\n'
           '    ret = await the_one_future\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 157, in run_pipeline_async\n'
           '    return await loop.run_in_executor(\n'
           '  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run\n'
           '    result = self.fn(*self.args, **self.kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 148, in '
           'run_pipeline_to_completion\n'
           '    raise e\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 134, in '
           'run_pipeline_to_completion\n'
           '    pipeline.run()\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 348, in run\n'
           '    for wu in itertools.islice(\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_source.py", line 1164, '
           'in get_workunits\n'
           '    ) = job.result()\n'
           '  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result\n'
           '    return self.__get_result()\n'
           '  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result\n'
           '    raise self._exception\n'
           '  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run\n'
           '    result = self.fn(*self.args, **self.kwargs)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_source.py", line 1049, '
           'in process_dashboard\n'
           '    metric_dim_workunits = self.process_metrics_dimensions_and_fields_for_dashboard(\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_source.py", line 880, '
           'in process_metrics_dimensions_and_fields_for_dashboard\n'
           '    chart_mcps = [\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_source.py", line 881, '
           'in <listcomp>\n'
           '    self._make_metrics_dimensions_chart_mcp(element, dashboard)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_source.py", line 988, '
           'in _make_metrics_dimensions_chart_mcp\n'
           '    fields=self._input_fields_from_dashboard_element(dashboard_element)\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_source.py", line 919, '
           'in _input_fields_from_dashboard_element\n'
           '    explore = self.explore_registry.get_explore(\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_common.py", line 882, '
           'in get_explore\n'
           '    looker_explore = LookerExplore.from_api(\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/looker_common.py", line 602, '
           'in from_api\n'
           '    from datahub.ingestion.source.looker.lookml_source import _BASE_PROJECT_NAME\n'
           '  File "/tmp/datahub/ingest/venv-looker-0.9.0/lib/python3.10/site-packages/datahub/ingestion/source/looker/lookml_source.py", line 11, '
           'in <module>\n'
           '    import lkml\n'
           "ModuleNotFoundError: No module named 'lkml'\n"
           '[2022-10-15 05:03:37,869] ERROR    {datahub.entrypoints:195} - Command failed: \n'
           "\tNo module named 'lkml'.\n"

It seems like

lkml

python package is missing for you :

"ModuleNotFoundError: No module named 'lkml'\n"

Can you install it manually? setup.py contains it, so normally it should install it -> https://github.com/datahub-project/datahub/blob/73fd35888b0a6de995ab2362cbcd65bab7095d5e/metadata-ingestion/setup.py#L275

@dazzling-judge-80093 Same error for me, I'm running the ingestion through UI. It seems that lkml dependency is listed on setup.py for

lookml

source but not for

looker

one

Hi there, I'm having the same issue : "ModuleNotFoundError: No module named 'lkml'\n". I have updated manually the lkml to 1.3.0b5 but still got the error when ingesting looker data. Any suggestions?

can you try to install the latest version (0.9.0.3) -> https://pypi.org/project/acryl-datahub/0.9.0.3/ This was released today and it should contain fix for this

I've installed it but the ingestion was still not working. In the end the error was fixed installing lkml gitpython and sqllineage in the venv-looker-0.9.0.

Hi there! See if one of the recommendations below helps you answer your question.