Hi, im seeing this error when trying to ingest fro...
# troubleshoot
l
Hi, im seeing this error when trying to ingest from BigQuery. Maybe someone knows something about that?
Copy code
~~~~ Execution Summary - RUN_INGEST ~~~~
Execution finished with errors.
{'exec_id': 'b7fe3905-6cb1-4c7b-9fb7-06f2e516826a',
 'infos': ['2023-04-18 19:28:43.814912 INFO: Starting execution for task with name=RUN_INGEST',
           "2023-04-18 19:28:49.881382 INFO: Failed to execute 'datahub ingest'",
           '2023-04-18 19:28:49.883130 INFO: Caught exception EXECUTING task_id=b7fe3905-6cb1-4c7b-9fb7-06f2e516826a, name=RUN_INGEST, '
           'stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 231, in execute\n'
           '    raise TaskError("Failed to execute \'datahub ingest\'")\n'
           "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"],
 'errors': []}

~~~~ Ingestion Logs ~~~~
Obtaining venv creation lock...
Acquired venv creation lock
venv setup time = 0
This version of datahub supports report-to functionality
datahub  ingest run -c /tmp/datahub/ingest/b7fe3905-6cb1-4c7b-9fb7-06f2e516826a/recipe.yml --report-to /tmp/datahub/ingest/b7fe3905-6cb1-4c7b-9fb7-06f2e516826a/ingestion_report.json
[2023-04-18 19:28:45,463] INFO     {datahub.cli.ingest_cli:173} - DataHub CLI version: 0.10.2
[2023-04-18 19:28:45,532] INFO     {datahub.ingestion.run.pipeline:204} - Sink configured successfully. DataHubRestEmitter: configured to talk to 
http://datahub-gms:8080
[2023-04-18 19:28:47,467] WARNING  {py.warnings:109} - /tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/ratelimiter.py:127: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
  __aexit__ = asyncio.coroutine(__exit__)

[2023-04-18 19:28:47,791] ERROR    {datahub.entrypoints:195} - Command failed: Failed to find a registered source for type bigquery: 'str' object is not callable
Traceback (most recent call last):
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 119, in _add_init_error_context
    yield
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 214, in __init__
    source_class = source_registry.get(source_type)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/api/registry.py", line 173, in get
    tp = self._ensure_not_lazy(key)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/api/registry.py", line 117, in _ensure_not_lazy
    plugin_class = import_path(path)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/api/registry.py", line 48, in import_path
    item = importlib.import_module(module_name)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 57, in <module>
    from datahub.ingestion.source.bigquery_v2.lineage import (
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/lineage.py", line 39, in <module>
    from datahub.utilities.bigquery_sql_parser import BigQuerySQLParser
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/utilities/bigquery_sql_parser.py", line 6, in <module>
    from datahub.utilities.sql_parser import SqlLineageSQLParser, SQLParser
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/utilities/sql_parser.py", line 9, in <module>
    from datahub.utilities.sql_lineage_parser_impl import SqlLineageSQLParserImpl
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/utilities/sql_lineage_parser_impl.py", line 8, in <module>
    from sqllineage.core.holders import Column, SQLLineageHolder
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/sqllineage/__init__.py", line 41, in <module>
    _monkey_patch()
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/sqllineage/__init__.py", line 35, in _monkey_patch
    _patch_updating_lateral_view_lexeme()
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/sqllineage/__init__.py", line 24, in _patch_updating_lateral_view_lexeme
    if regex("LATERAL VIEW EXPLODE(col)"):
TypeError: 'str' object is not callable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/entrypoints.py", line 182, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 379, in wrapper
    raise e
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 334, in wrapper
    res = func(*args, **kwargs)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/utilities/memory_leak_detector.py", line 95, in wrapper
    return func(ctx, *args, **kwargs)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 187, in run
    pipeline = Pipeline.create(
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 328, in create
    return cls(
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 211, in __init__
    with _add_init_error_context(
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/tmp/datahub/ingest/venv-bigquery-0.10.2/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 121, in _add_init_error_context
    raise PipelineInitError(f"Failed to {step}: {e}") from e
datahub.ingestion.run.pipeline.PipelineInitError: Failed to find a registered source for type bigquery: 'str' object is not callable
DataHub CLI version: 0.10.2 Python version: 3.10.10 (main, Feb 8 2023, 053204) [Clang 14.0.0 (clang-1400.0.29.202)]
đź“– 1
👍 2
🔍 1
âś… 2
l
This content can't be displayed.
n
@lively-businessperson-84910 — were you able to find a solution? I’ve been dealing with the same issue with the “redshift” source too
l
@nice-rose-9274 Unfortunately not, i’ll keep you posted here if i find a solution
b
Same with metabase:
datahub.ingestion.run.pipeline.PipelineInitError: Failed to find a registered source for type metabase: 'str' object is not callable
l
hmm, still no progress from my end
f
Facing the same issue with Redshift like @nice-rose-9274 mentioned. Was working 2 days ago before the update so not sure what broke. Are there project experts here who might be able to help?
Looking at this a little further this seems to be tied to the sqllineage module. Searched a bit deeper Ref: https://github.com/reata/sqllineage/issues/361 and it looks like was a bug that came about from the latest release of the dependent sqlparse module. Comment notes that a new PR to fix this is incoming (no ETA)
f
We are having this same issue in Redshift. Two days ago 0.10.1 worked and now our new build does not work:
Copy code
powerbi_1  |     from sqllineage.runner import LineageRunner
powerbi_1  |   File "/usr/local/lib/python3.10/site-packages/sqllineage/__init__.py", line 41, in <module>
powerbi_1  |     _monkey_patch()
powerbi_1  |   File "/usr/local/lib/python3.10/site-packages/sqllineage/__init__.py", line 35, in _monkey_patch
powerbi_1  |     _patch_updating_lateral_view_lexeme()
powerbi_1  |   File "/usr/local/lib/python3.10/site-packages/sqllineage/__init__.py", line 24, in _patch_updating_lateral_view_lexeme
powerbi_1  |     if regex("LATERAL VIEW EXPLODE(col)"):
powerbi_1  | TypeError: 'str' object is not callable
powerbi_1  |
powerbi_1  | The above exception was the direct cause of the following exception:
powerbi_1  |
powerbi_1  | Traceback (most recent call last):
powerbi_1  |   File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
powerbi_1  |     return _run_code(code, main_globals, None,
powerbi_1  |   File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
powerbi_1  |     exec(code, run_globals)
powerbi_1  |   File "/usr/local/app/src/yle/datahub/data_loader.py", line 101, in <module>
powerbi_1  |     main()
powerbi_1  |   File "/usr/local/app/src/yle/datahub/data_loader.py", line 85, in main
powerbi_1  |     pipeline = Pipeline.create(pipeline_config)
powerbi_1  |   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 308, in create
powerbi_1  |     return cls(
powerbi_1  |   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 191, in __init__
powerbi_1  |     with _add_init_error_context(
powerbi_1  |   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
powerbi_1  |     self.gen.throw(typ, value, traceback)
powerbi_1  |   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 117, in _add_init_error_context
powerbi_1  |     raise PipelineInitError(f"Failed to {step}: {e}") from e
powerbi_1  | datahub.ingestion.run.pipeline.PipelineInitError: Failed to find a registered source for type redshift: 'str' object is not callable
@famous-barista-37580 found the root cause. Sqllineage is not pinning versions of sqlparse & sqlfluff. --> Datahub should update to new sqllineage dependency when new release is available.
We have a separate loader application, which uses acryl-datahub -package. As a workaround you can pin working versions of dependencies to your wrapper application's requirements.txt:
Copy code
acryl-datahub==X.X:X
...
sqlparse==0.4.3
sqlfluff==2.0.3
a
I have the same case with LookML
f
r
I see it's been fixed in the following PRs https://github.com/datahub-project/datahub/pull/7847 https://github.com/datahub-project/datahub/pull/7848 but there hasn't been a release yet
is there a way to pin the versions for ui ingestion?
so to correct my earlier message, the fix was actually released yesterday in
0.10.2.1
. I use UI ingestion so changing
UI_INGESTION_DEFAULT_CLI_VERSION
to `0.10.2.1` in datahub-gms resolves the issue for me
h
i was running datahub quick start and still facing this issues.How can i fix this
when big query install package it still install sqlparse 0.4.4 and acryl datahub is still 0.10.2 in UI
r
@refined-energy-76018 appreciate that if show the step in detail. does it apply to
datahub docker quickstart
?
r
I don't use quickstart but I think you can use a custom docker file like here and add
- UI_INGESTION_DEFAULT_CLI_VERSION=0.10.2.1
under
datahub-gms: environment:
h
i just change the ui cli to 0.10.2.1 and it work
f
This really works! thank you
a
I change the UI cli to 0.10.2.1 but still get this error, does anyone have same issue or could figure out how to fix it?
h
@ancient-guitar-42996 same. The first time after i change the ui cli it worked , but next time i ingest it return error :
a
@gray-shoe-75895 / @gentle-hamburger-31302 - would this have been fixed by the recent patch?
a
Yes this should be fixed by the patch release. We’d recommend using 0.10.2.2, since it contains a few other important patches
c
@gray-shoe-75895 Is it visible anywhere which commits made it into 0.10.2.1 and 0.10.2.2?
g
Yup - we cut the cli releases from this repo https://github.com/acryldata/datahub/releases, and the full changelog links will have the lists of commits
c
great, thanks a lot!
l
I have set the
UI_INGESTION_DEFAULT_CLI_VERSION=0.10.2.2
in dockerfile and start it, but the CLI version still be 0.10.2.1, anyone know how to fix it?
g
Hi @late-smartphone-6255 fyi... you can override cli version while creating managed ingestion on DataHub Portal. Steps are mentioned at https://datahubproject.io/docs/ui-ingestion/#advanced-running-with-a-specific-cli-version
l
Thank you @gentle-hamburger-31302, I try that way but still not work
a
It worked for me , I was getting same error then I set the version while creating managed ingestion and it worked
f