Is there a way to disable the Airflow Datahub plugin locally DataHub #ingestion

Is there a way to disable the Airflow Datahub plug...

refined-energy-76018

09/29/2022, 3:35 AM

Is there a way to disable the Airflow Datahub plugin locally which doesn't involve removing the dependency? For example, if I use the lineage backend, I can just inject

AIRFLOW__LINEAGE__BACKEND

set to an empty string to 'disable' the integration. When I set

AIRFLOW__DATAHUB__CONN_ID

to an empty string or remove the

[datahub]

configuration from

airflow.cfg

entirely, I still get the error messages in the task logs which is noisy.

gray-shoe-75895

09/29/2022, 4:56 PM

Just added an

enabled

flag to the config https://github.com/datahub-project/datahub/pull/6089 🙂

refined-energy-76018

09/29/2022, 5:52 PM

thanks, this looks awesome!

gray-shoe-75895

11/22/2022, 10:55 PM

Taking a guess at what could be wrong: your config should be in airflow.cfg under datahub, not under lineage

Copy code

[datahub]
enabled=false

gray-shoe-75895

11/22/2022, 10:56 PM

The lineage piece was from the lineage backend, which we used before we developed the plugin version

refined-energy-76018

11/22/2022, 10:57 PM

message has been deleted

refined-energy-76018

11/22/2022, 10:59 PM

@gray-shoe-75895 do you know if this check should be in this file?

refined-energy-76018

11/22/2022, 11:02 PM

when I run a test DAG with the flag set, I see the emit happening from the

_plugin.py

file

gray-shoe-75895

11/23/2022, 5:52 AM

Yup you’re right - fixing it here https://github.com/datahub-project/datahub/pull/6528

refined-energy-76018

11/23/2022, 6:06 AM

Awesome! Thanks for the quick turnaround

5 Views

Open in Slack

Previous Next