A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

<@U01GCJKA8P9> You mentioned in a YT video that the lineage supports redshift dataset now. Is it under the latest release?

Hi <@U02Q6ASV600>! Jumping in for John - we’re still putting some finishing touches on the Redshift dataset lineage; targeting release within the next 1-2wks (cc <@UV14447EU>)

qq, I’m just curious how do we generate dataset lineage for Redshift? is it generated from its own access logs like it’s handled in Snowflake?

There are multiple way to do it:
For queries:
1. Using stl_insert (<https://docs.aws.amazon.com/redshift/latest/dg/r_STL_INSERT.html>) to find tables where inster happened and using stl_scan (<https://docs.aws.amazon.com/redshift/latest/dg/r_STL_SCAN.html>) for finding upstream tables -&gt; works well except for Spectrum tables
2. Doing query parsing based on stl_insert (slower than above and less reliabel but works for Spectrum as well)

There are some other cases like view/non-binding views lineage and copy commands which we also would like to support

Thank you <@UV14447EU> for detailed explanation! I appreciate that :)