https://datahubproject.io logo
Join Slack
Powered by
Hi, I’ve upgrade my instance of datahub `0.8.26` -...
# troubleshoot
b
Hi, I’ve upgrade my instance of datahub 
0.8.26
-> 
0.8.32.1
and ingestion library 
0.8.22.1
-> 
0.8.32.1
and now I am getting errors when I try to ingest data from looker:
Copy code
TypeError: You should use `typing_extensions.TypedDict` instead of `typing.TypedDict` with Python < 3.9.2. Without it, there is no way to differentiate required and optional fields when subclassed.
full stack trace in thread. Any ideas?
Copy code
[2022-04-05 13:19:57,481] INFO     {datahub.ingestion.sink.datahub_rest:60} - Setting gms config
[2022-04-05 13:19:57,481] DEBUG    {datahub.telemetry.telemetry:58} - Updating telemetry config
[2022-04-05 13:19:57,482] DEBUG    {datahub.ingestion.run.pipeline:123} - Sink type:datahub-rest,<class 'datahub.ingestion.sink.datahub_rest.DatahubRestSink'> configured
[2022-04-05 13:19:57,994] DEBUG    {datahub.telemetry.telemetry:159} - Sending Telemetry
[2022-04-05 13:19:58,321] ERROR    {datahub.entrypoints:152} - File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 82, in run
    70   def run(
    71       ctx: click.Context, config: str, dry_run: bool, preview: bool, strict_warnings: bool
    72   ) -> None:
 (...)
    78       pipeline_config = load_config_file(config_file)
    79
    80       try:
    81           logger.debug(f"Using config: {pipeline_config}")
--> 82           pipeline = Pipeline.create(pipeline_config, dry_run, preview)
    83       except ValidationError as e:

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 175, in create
    171  def create(
    172      cls, config_dict: dict, dry_run: bool = False, preview_mode: bool = False
    173  ) -> "Pipeline":
    174      config = PipelineConfig.parse_obj(config_dict)
--> 175      return cls(config, dry_run=dry_run, preview_mode=preview_mode)

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 126, in __init__
    105  def __init__(
    106      self, config: PipelineConfig, dry_run: bool = False, preview_mode: bool = False
    107  ):
 (...)
    122      self.sink: Sink = sink_class.create(sink_config, self.ctx)
    123      logger.debug(f"Sink type:{self.config.sink.type},{sink_class} configured")
    124
    125      source_type = self.config.source.type
--> 126      source_class = source_registry.get(source_type)
    127      self.source: Source = source_class.create(

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 126, in get
    115  def get(self, key: str) -> Type[T]:
 (...)
    122
    123      if key not in self._mapping:
    124          raise KeyError(f"Did not find a registered class for {key}")
    125
--> 126      tp = self._ensure_not_lazy(key)
    127      if isinstance(tp, ModuleNotFoundError):

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 84, in _ensure_not_lazy
    80   def _ensure_not_lazy(self, key: str) -> Union[Type[T], Exception]:
    81       path = self._mapping[key]
    82       if isinstance(path, str):
    83           try:
--> 84               plugin_class = import_path(path)
    85               self.register(key, plugin_class, override=True)

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 32, in import_path
    18   def import_path(path: str) -> Any:
 (...)
    28           module_name, object_name = path.rsplit(":", 1)
    29       else:
    30           module_name, object_name = path.rsplit(".", 1)
    31
--> 32       item = importlib.import_module(module_name)
    33       for attr in object_name.split("."):

File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    109  def import_module(name, package=None):
 (...)
    123          for character in name:
    124              if character != '.':
    125                  break
    126              level += 1
--> 127      return _bootstrap._gcd_import(name[level:], package, level)

File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

File "<frozen importlib._bootstrap>", line 991, in _find_and_load

File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 671, in _load_unlocked

File "<frozen importlib._bootstrap_external>", line 783, in exec_module

File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/looker.py", line 66, in <module>
    62
    63   logger = logging.getLogger(__name__)
    64
    65
--> 66   class LookerAPIConfig(ConfigModel):
    67       client_id: str

File "pydantic/main.py", line 204, in pydantic.main.ModelMetaclass.__new__

File "pydantic/fields.py", line 488, in pydantic.fields.ModelField.infer

File "pydantic/fields.py", line 419, in pydantic.fields.ModelField.__init__

File "pydantic/fields.py", line 539, in pydantic.fields.ModelField.prepare

File "pydantic/fields.py", line 801, in pydantic.fields.ModelField.populate_validators

File "pydantic/validators.py", line 696, in find_validators

File "pydantic/validators.py", line 585, in pydantic.validators.make_typeddict_validator

File "pydantic/annotated_types.py", line 24, in pydantic.annotated_types.create_model_from_typeddict

---- (full traceback above) ----
File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 82, in run
    pipeline = Pipeline.create(pipeline_config, dry_run, preview)
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 175, in create
    return cls(config, dry_run=dry_run, preview_mode=preview_mode)
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 126, in __init__
    source_class = source_registry.get(source_type)
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 126, in get
    tp = self._ensure_not_lazy(key)
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 84, in _ensure_not_lazy
    plugin_class = import_path(path)
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/api/registry.py", line 32, in import_path
    item = importlib.import_module(module_name)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/looker.py", line 66, in <module>
    class LookerAPIConfig(ConfigModel):
File "pydantic/main.py", line 204, in pydantic.main.ModelMetaclass.__new__
File "pydantic/fields.py", line 488, in pydantic.fields.ModelField.infer
File "pydantic/fields.py", line 419, in pydantic.fields.ModelField.__init__
File "pydantic/fields.py", line 539, in pydantic.fields.ModelField.prepare
File "pydantic/fields.py", line 801, in pydantic.fields.ModelField.populate_validators
File "pydantic/validators.py", line 696, in find_validators
File "pydantic/validators.py", line 585, in pydantic.validators.make_typeddict_validator
File "pydantic/annotated_types.py", line 24, in pydantic.annotated_types.create_model_from_typeddict

TypeError: You should use `typing_extensions.TypedDict` instead of `typing.TypedDict` with Python < 3.9.2. Without it, there is no way to differentiate required and optional fields when subclassed.
[2022-04-05 13:19:58,322] INFO     {datahub.entrypoints:161} - DataHub CLI version: 0.8.30.0 at /usr/local/lib/python3.8/site-packages/datahub/__init__.py
[2022-04-05 13:19:58,322] INFO     {datahub.entrypoints:164} - Python version: 3.8.9 (default, Apr 10 2021, 15:55:09)
[GCC 8.3.0] at /usr/local/bin/python on Linux-5.10.76-linuxkit-x86_64-with-glibc2.2.5
[2022-04-05 13:19:58,322] INFO     {datahub.entrypoints:167} - GMS config {'models': {}, 'versions': {'linkedin/datahub': {'version': 'v0.8.32', 'commit': 'ede6547eff1496a87048e5520f1d9e53e148f72c'}}, 'managedIngestion': {'defaultCliVersion': '0.8.26.6', 'enabled': True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, 'retention': 'true', 'noCode': 'true'}
I’ve deduced that this starts with version 
0.8.30
of the ingestion library.
But still works fine with 
0.8.29
3 Views
Open in Slack
PreviousNext
Powered by