dazzling-judge-80093
04/12/2022, 7:06 PMdazzling-judge-80093
04/12/2022, 7:06 PMdelightful-barista-90363
04/12/2022, 7:07 PMdazzling-judge-80093
04/12/2022, 7:08 PMdazzling-judge-80093
04/12/2022, 7:10 PMSoftware-defined assets vs. Asset Materializations#
When working with software-defined assets, the assets and their dependencies must be known at definition time. When you look at software-defined assets in Dagit, you can see exactly what assets are going to be materialized before any code runs.
Asset Materializations, on the other hand, are logged at run time. When you run an op, you find out which assets were materialized while the op is running. This allows for some flexibility, like if you wanted to determine which assets should be materialized based on the output of a previous op.
dazzling-judge-80093
04/12/2022, 7:11 PMdazzling-judge-80093
04/12/2022, 7:12 PMdelightful-barista-90363
04/12/2022, 7:13 PMdelightful-barista-90363
04/12/2022, 7:15 PMdazzling-judge-80093
04/12/2022, 7:17 PMdazzling-judge-80093
04/12/2022, 7:17 PMdelightful-barista-90363
04/12/2022, 7:18 PMdelightful-barista-90363
04/12/2022, 7:19 PMdelightful-barista-90363
04/12/2022, 7:24 PMdazzling-judge-80093
04/12/2022, 7:40 PMdelightful-barista-90363
04/12/2022, 7:45 PMdelightful-barista-90363
04/12/2022, 7:46 PMdelightful-barista-90363
04/27/2022, 6:10 PMdelightful-barista-90363
04/28/2022, 5:57 PM• Datahub Dagster Integration
◦ Dagster control datahub ingest jobs (similar to https://datahubproject.io/docs/metadata-ingestion/schedule_docs/airflow)
◦ Datahub @resource for dagster (for pushing MCE)
◦ Datahub pull lineage from Dagster
◦ Datahub Pull metadata from dagster
◦ Datahub Dagster (Task) links to dagster job
dazzling-judge-80093
04/28/2022, 5:59 PMdelightful-barista-90363
04/28/2022, 6:00 PMdelightful-barista-90363
04/29/2022, 2:32 PMdazzling-judge-80093
04/29/2022, 2:33 PMdazzling-judge-80093
04/29/2022, 2:34 PMdelightful-barista-90363
04/29/2022, 2:35 PMrapid-king-93225
05/04/2022, 10:38 AMdelightful-barista-90363
07/06/2022, 5:36 PMalert-alligator-12918
11/30/2022, 11:17 PMstocky-monkey-23735
08/08/2023, 1:24 AMbulky-shoe-65107
10/16/2023, 12:37 AM