I’m/was working recently on building some easier abstraction capturing DataJob (Op)/DataFlow (Pipeline) And there is a new entity DataProcessInstance to capture actual pipeline run/OP run.

I havent worked with software defined assets, but that seems like something that would better integrate with datahub then some older dagster features

Yeah, I need to check it out as so far I was thinking in AssetMaterialization/capturing it from IOManager

It seems like this is the difference between the two:

Software-defined assets vs. Asset Materializations#
When working with software-defined assets, the assets and their dependencies must be known at definition time. When you look at software-defined assets in Dagit, you can see exactly what assets are going to be materialized before any code runs.

Asset Materializations, on the other hand, are logged at run time. When you run an op, you find out which assets were materialized while the op is running. This allows for some flexibility, like if you wanted to determine which assets should be materialized based on the output of a previous op.

which seems like Software defined asset is an expectation and Asset Materialization is the reality

actually we can support both with the new model because you can define dependencies with every run and this way you will be able to see the graph/inputs/outputs per run

Thats another feature/question that I had with my team. Being how granular should lineage get wrt graphs, jobs, and ops

Theres also the question of how to best integrate external resources, as theres usually a lot of custom code (at least in my use case) surrounding an S3 or Postgres Resource for example

do you expose asset materializations?

then I think you don’t have to touch any custom code/resource

we do

i was referring more to inputs being a separate s3 bucket or database. We generally have a raw data bucket that gets run through transformations into separate s3 buckets

Assets i do agree could get by without custom code, and as a user, probably would want to only track lineage for assets that are materialized

Are you saying in your resources you are doing transformations or the problem is more that how to capture those data which is not generated in Dagster but more before that?

Capture data not generated in Dagster

for example: download files from an s3 bucket, run transformations that yield assets to a separate bucket. Pretty much creating the connection from the source bucket -> dagster job -> target bucket

Hey, I got some room to discuss and work on a dagster integration now

Some initial ideas i had were

• Datahub Dagster Integration

◦ Dagster control datahub ingest jobs (similar to https://datahubproject.io/docs/metadata-ingestion/schedule_docs/airflow)

◦ Datahub @resource for dagster (for pushing MCE)

◦ Datahub pull lineage from Dagster

◦ Datahub Pull metadata from dagster

◦ Datahub Dagster (Task) links to dagster job

Hey, if you have time we can catch up about this tomorrow

sounds good to me

gonna bump this and see if you wanna meet today

When are you available?

I will have one meeting 1 hour from now but we can have some discussion before or after that

im free all day

do not forget their more rich concept around assets

For a small integration with dagster + datahub, i made the Rest and Kafka emitters available as Dagster Resources. This isn't the full blown "metadata exchange" between the two services. That will require a lot more work and time lol. Here is the PR for said work. https://github.com/dagster-io/dagster/pull/8764

Hey! I was curious if anyone can point towards an example of this integration being used 🙂

I saw this and got curious. Do we have progress related to this? I'm in an organization where we also have to implement this. We're using Datahub and Dagster along with DBT.

has renamed the channel from "integration-dagster-datahub" to "integrate-dagster-datahub"