With MWAA Supporting Airflow version 2.2.2 has any one in the community tried to push the lineage from MWAA to Datahub? https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-versions.html#airflow-versions-v222

I know @dazzling-judge-80093 has been doing some research into supporting managed airflow!

@dazzling-judge-80093 Any inputs on MWAA -> Datahub would be great.

I could install Datahub airflow on the latest MWAA and airflow 1 as well but I need to check which version I installed exactly. I will get back to you.

Steps followed: 1. Added

acryl-datahub[airflow]

plugin to the requirements.txt 2. Added a connection

datahub-rest-default

to GMS 3. Added the following configs:

lineage.backend

-

datahub_provider.lineage.datahub.DatahubLineageBackend

lineage.datahub_kwargs

-

{\"datahub_conn_id\": \"datahub_rest_default\",\"cluster\": \"dev\",\"capture_ownership_info\": true,\"capture_tags_info\": true,\"graceful_exceptions\": true }

After adding the configs the cluster update fails without any details on errors. Will update as i learn more.

I used the Lineage Backend for my demo which capture inlets/outlets and lineage as well -> https://datahubproject.io/docs/lineage/airflow Did you used the LineageBackend as well?

Thanks for the response! Yes, I used the lineage backend as well but that didn't work there were no lineages pushed from MWAA. So had to resort to using

DatahubEmitterOperator

The hard part is MWAA lineage backend doesn't have much logs too...

I work that part currently and I will make sure to improve logging the backend

@nutritious-bird-77396 did you actually have to escape your json for lineage.datahub_kwargs config?

@dazzling-judge-80093 Its not about Datahub logging...its about MWAA lineage backend logging

ahh, I see, I know it is horrible

@damp-greece-27806 I had to add it as an env variable in the DAG for now...

os.environ["AIRFLOW__LINEAGE__BACKEND"] = "datahub_provider.lineage.datahub.DatahubLineageBackend"
os.environ[
    "AIRFLOW__LINEAGE__DATAHUB_KWARGS"
] = '{"datahub_conn_id":"datahub_rest","cluster":"dev","capture_ownership_info":true,"capture_tags_info":true,"graceful_exceptions":true}'

Hmm that is unfortunate

I reached out AWS support regarding it and they said that MWAA doesn’t support lineage as it’s an experimental feature.

Thanks, good to know it