Hello i am trying to connect airflow to datahub 0....
# troubleshoot
p
Hello i am trying to connect airflow to datahub 0.9.1 , I am new to airflow and done till step 4(adding airflow hook) mentioned here- https://datahubproject.io/docs/lineage/airflow/ but I am not seeing anything on the airflow> admin> plugins. Can someone help with what could be going wrong here?
m
Do you see the connection in Admin>Connections ?? Have you set the conn id to the names suggested and not others? And finally, have you downloaded the required plugin in the webserver and workers (and it also might be needed in the scheduler although I am not 100% sure)? As I guess that your Airflow's version is greater than 2.0.2, right?
p
Hey @microscopic-mechanic-13766 I can see the conn Id in connection list, it is the same name as suggested in the document. Not sure of the required plugin, can u share some document around this setup. And yes airflow version is 2.4.2
m
The plugin is the one mentioned in the point 1:
pip install acryl-datahub-airflow-plugin
I haven't tested myself the 0.9.1 version of the plugin but the version 0.8.45 works fine
p
Yes the plugin is installed
m
Do the logs show any error?? I also have the following properties:
Copy code
AIRFLOW__LINEAGE__BACKEND: datahub_provider.lineage.datahub.DatahubLineageBackend
        AIRFLOW__LINEAGE__DATAHUB_CONN_ID: 'datahub_rest_default'
        AIRFLOW__LINEAGE__CAPTURE_OWNERSHIP_INFO: 'True'
        AIRFLOW__LINEAGE__CAPTURE_TAGS_INFO: 'True'
        AIRFLOW__LINEAGE__GRACEFUL_EXCEPTIONS: 'True'
        AIRFLOW__LINEAGE__DATAHUB_KWARGS: '{"capture_executions": true}'
Note that these properties are not the exactly the same as the ones you would have to writte in the airflow.cfg file, they would have to be without "AIRFLOW__LINEAGE__" if I am not mistaken
p
Yes they are without AIRFLOW_LINEAGE and present in airflow cfg. No error on logs
m
This may sound a bit basic, but have you tried restarting airflow? I don't see any problem in your configuration at all.
p
Let me try once again
f
@polite-ghost-91039 How did you manage to resolve this issue