Yea we do @ classdojo; AMA
# randomd
damp-greece-27806
03/23/2022, 12:44 PMYea we do @ classdojo; AMA
d
dazzling-judge-80093
03/23/2022, 2:55 PMDo you use mwaa and datahub with the lineage backend?
d
damp-greece-27806
03/23/2022, 2:56 PMyes
damp-greece-27806
03/23/2022, 2:56 PMwith datahub lib version 0.8.31
d
dazzling-judge-80093
03/23/2022, 6:43 PMCan you share how you set it up?
d
damp-greece-27806
03/23/2022, 6:55 PMsure, there’s not much to it; we use the python library as opposed to adding an entrypoint to call the CLI. We use the pipeline class to write recipes inline as JSON
damp-greece-27806
03/23/2022, 6:59 PM@dazzling-judge-80093 is there anything specific about the setup?
damp-greece-27806
03/23/2022, 6:59 PMthat you’d be interested in hearing more about
damp-greece-27806
03/23/2022, 7:03 PMoh, sorry, you specifically asked about the lineage backend
damp-greece-27806
03/23/2022, 7:28 PMwe use the environment variables provided in the datahub documentation:
Copy code
AIRFLOW__LINEAGE__BACKEND=datahub_provider.lineage.datahub.DatahubLineageBackend
AIRFLOW__LINEAGE__DATAHUB_KWARGS={"datahub_conn_id":"datahub"}damp-greece-27806
03/23/2022, 7:28 PMand then we use AUTO for the airflow inlets, and specify our outlet dataset
damp-greece-27806
04/04/2022, 7:34 PMjust a FYI on this - MWAA doesn’t seem to support the lineage config, at least not on 2.0.2; I’m trying what someone else suggested (use env vars in your DAG), but i’m not convinced it’s going to work in production based on local testing with the mwaa runner
2 Views