Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hey everyone :wave:  I’m here to answer any questions you might have during the session :slightly_smiling_face:

And I’m here too, tag-teaming with <@U04EG7DV3R7>. Drop us a note about anything :wave:

Thanks for watching, folks! If you’re interested in learning more, here are a couple of our free resources about active metadata, DataOps, and the future of data catalogs :rocket:

<https://atlan.com/resources/data-catalog-primer/>
<https://atlan.com/resources/dataops-active-metadata-report/>

Very interesting talk. I'm struggling to conceptualize the relationship between orchestration tools and active metadata - they clearly have different areas of strength/focus but also need to work together (I'd think). Is there already a good resource that dives deeper into this?

Hey <@U03FNJGED7D> – let me dig up some content we can share on this.

I absolutely agree that they need to work together. The orchestration tools play a key role in orchestrating the movement of data and the journey it takes

In Atlan’s case, a lot of our customers push that type of operational metadata into the catalog also. For example if a dbt run was triggered via an orchestration tool, push the metadata of that job into Atlan can provide some rich metadata to bring more context to an asset, e.g. a table in Snowflake

To be more specific, we're users of software defined assets in dagster, which has a great data lineage graph (with some metadata). I think Atlan can potentially supplement it.

Ah okay! Yeah, we will also likely look to integrate with Dagster in the future

Lineage is quite interesting, because looking at lineage inside dbt or dagster can often be very data analyst focused (dbt) or data engineer focused (dagster)

The lineage Atlan provides brings that whole lineage flow together, with a focus on what it will be used for

Yeah! I think that's a good description for what I'm trying to think about. We already have both the dbt lineage graph + docs and dagster's assets and lineage. those seem to complement each other, so it makes sense to me to deploy both.

A simple example here: Lineage within Snowflake based on all the transformations triggered by dbt, and downstream into the BI layer to dashboards/reports etc…

The value of this lineage graph is the documentation (catalog) behind each asset in the lineage

For example say you’re thinking about impact analysis (updating schema of a Postgres DB). The graph can show you everything you will impact downstream based on this change, who the owners of those assets are, etc…

Or vice versa, doing root cause analysis to identify why a BI dashboard broke. This would allow you to navigate upstream, identify everything you need to know about the asset (e.g. Who owns that dbt model, or Snowflake view)

Here’s an example of what that asset could look like, if we wanted to dig into a specific asset from the lineage graph

In this scenario we’ve even got announcement on the asset because of a failed Airflow DAG. A similar scenario could exist with Dagster

Here’s some info on active metadata specifically: <https://humansofdata.atlan.com/2022/05/what-is-active-metadata/>

Here’s another way to think of it. The Data Catalog provides visibility over all the assets in your data stack. The data catalog is then enriched with additional metadata from dbt, dagster, even data quality tools like Monte Carlo. This gives you all the context about your asset under one roof

<@U03FNJGED7D> this blog post is actually much better: <https://humansofdata.atlan.com/2022/05/what-is-active-metadata/>