Hello everybody. So, I'm trying to ingest information from airflow. Not only the pipelines, but mainly to create lineage. I've followed this guide https://datahubproject.io/docs/lineage/airflow/. I do have the plugin installed and the connection is set. I've also used https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub_provider/example_dags/lineage_backend_demo.py this lineage demo to try to test it. I though this was enough, but then I read that the dag itself does nothing, it needs to establish the connection that should be exposed on the docker compose file. So I added this that was on other question:

AIRFLOW_CONN_DATAHUB_REST_DEFAULT: <datahub-rest://http>%3A%2F%2Fdatahub-gms%3A8080
AIRFLOW__LINEAGE__BACKEND: datahub_provider.lineage.datahub.DatahubLineageBackend
AIRFLOW__LINEAGE__DATAHUB_KWARGS: '{"datahub_conn_id": "datahub_rest_default",
                                 "capture_ownership_info": true,
                                 "capture_tags_info": true,
                                 "graceful_exceptions": true }'

Once I add this to my docker compose file (is the one airflow provides, no changes) I can't get the airflow to start. Could you please help me? Thank you

Are you saying you are not seeing dags/tasks and inlets/outlets in Datahub? Do you see anything in your task logs?

Yes, exactly. My task log run. The only thing I see there is the bash command executing echo

Which version of airflow do you have? Which datahub package you installed?

So airflow 2.3.2 and i've installed acryl-datahub 0.8.40.2 I'm now getting an error saying no module named datahub_provider..

If you run Airflow and Datahub as well from Docker then you have to make sure your airflow can access datahub. Similar like here: https://github.com/datahub-project/datahub/blob/558a65a3c32e3b1f7d4ec9a6e3848275b75a9656/docker/airflow/docker-compose.yaml#L72

omg no way, I was missing that! And it makes a lot of sense!! Thank you so much!!!