Hi, The airflow integration was working fine. I a...
# all-things-deployment
a
Hi, The airflow integration was working fine. I am getting the below error suddenly. What might have caused this?
Copy code
[2022-09-06, 14:55:50 UTC] {base.py:68} INFO - Using connection ID 'datahub_rest_default' for task execution.
[2022-09-06, 14:55:50 UTC] {logging_mixin.py:115} INFO - Exception: Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_airflow_plugin/datahub_plugin.py", line 318, in custom_on_failure_callback
    datahub_on_failure_callback(context)
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_airflow_plugin/datahub_plugin.py", line 125, in datahub_on_failure_callback
    DatahubGenericHook(context["_datahub_config"].datahub_conn_id)
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/hooks/datahub.py", line 77, in make_emitter
    return datahub.emitter.rest_emitter.DatahubRestEmitter(*self._get_config())
AttributeError: module 'datahub' has no attribute 'emitter
d
Is this a one-time issue? As I can see, it happened on a failure. It seems like it can find the datahub module somehow
a
no, it is happening for all the DAGs for every run... Not sure what happened. It was working fine before.
d
Can you try reinstalling the datahub package, it seems like some part of it got removed or I don’t know but emitter should be there: https://github.com/datahub-project/datahub/tree/master/metadata-ingestion/src/datahub/emitter
a
Copy code
[2022-09-07, 00:15:29 UTC] {base.py:68} INFO - Using connection ID 'datahub_rest_default' for task execution.
[2022-09-07, 00:15:29 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1451, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1601, in _execute_task_with_callbacks
    self.task.post_execute(context=context, result=result)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/lineage/__init__.py", line 124, in wrapper
    _backend.send_lineage(operator=self, inlets=self.inlets, outlets=self.outlets, context=context)
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/lineage/datahub.py", line 117, in send_lineage
    send_lineage_to_datahub(
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/_lineage_core.py", line 58, in send_lineage_to_datahub
    emitter = hook.make_emitter()
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/hooks/datahub.py", line 217, in make_emitter
    return self.get_underlying_hook().make_emitter()
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/hooks/datahub.py", line 77, in make_emitter
    return datahub.emitter.rest_emitter.DatahubRestEmitter(*self._get_config())
AttributeError: module 'datahub' has no attribute 'emitter'
For some reason it is trying to use the datahub_provider/hooks/datahub.py path