https://datahubproject.io logo
Join Slack
Powered by
Hey team, I got the following error when running L...
# troubleshoot
r
Hey team, I got the following error when running Lookml ingestion pipeline
Copy code
You should use `typing_extensions.TypedDict` instead of `typing.TypedDict` with Python < 3.9.2. Without it, there is no way to differentiate required and optional fields when subclassed.
related stack track
Copy code
File "python/tasks/datahub/looker/lookml_metadata_export_task.py", line 79, in _build_datahub_pipeline
    return Pipeline.create(
  File ".ijwb/venv/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 184, in create
    return cls(
  File ".ijwb/venv/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 131, in __init__
    source_class = source_registry.get(source_type)
  File ".ijwb/venv/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 126, in get
    tp = self._ensure_not_lazy(key)
  File ".ijwb/venv/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 84, in _ensure_not_lazy
    plugin_class = import_path(path)
  File ".ijwb/venv/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 32, in import_path
    item = importlib.import_module(module_name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File ".ijwb/venv/lib/python3.8/site-packages/datahub/ingestion/source/lookml.py", line 49, in <module>
    from datahub.ingestion.source.looker import LookerAPI, LookerAPIConfig
  File ".ijwb/venv/lib/python3.8/site-packages/datahub/ingestion/source/looker.py", line 66, in <module>
    class LookerAPIConfig(ConfigModel):
  File "pydantic/main.py", line 204, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 488, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 419, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 539, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 801, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 696, in find_validators
  File "pydantic/validators.py", line 585, in pydantic.validators.make_typeddict_validator
  File "pydantic/annotated_types.py", line 24, in pydantic.annotated_types.create_model_from_typeddict
TypeError: You should use `typing_extensions.TypedDict` instead of `typing.TypedDict` with Python < 3.9.2. Without it, there is no way to differentiate required and optional fields when subclassed.
b
yeah I have the same issue - would be a shame if we needed python 3.9 - I am sort of bound to 3.8 at the moment.
b
@helpful-optician-78938 @dazzling-judge-80093 Please advise on this issue - Do folks need > 3.9 in general? For just this source?
If yes, we need to add a disclaimer 🙂
d
It shouldn't but other users reported this issue as well. I will try to test with real looker as this issue doesn't happen with our integration tests
b
I'm also getting the exact same issue for ingesting Looker and LookML after upgrading DataHub from 
0.8.24 -> 0.8.31.4
. Upgrading python to 3.9.2 solves the issue, but was working fine using 3.8 previously which we're currently heavily tied too.
b
Can confirm that works with python 3.9.12
d
I will try to reproduce and fix this issue tomorrow
thank you 2
2 Views
Open in Slack
PreviousNext
Powered by