A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

<@U028JK2U492> <@U01C3DMG2GL> following up our discussion in the townhall zoom  chat. We’ve built a custom ingestion module to map table-&gt;table or table-&gt;view or basically object-&gt;object dependency in redshift. This is the only rule where we have a dataset to dataset mapping. Most cases its dataset -&gt; airflow/databricks -&gt; dataset.
I can write something up but its fairly simple. We have our own SQL parser to capture *view-&gt;view* dependency, but if the upstream dataset is a table then you can use redshift’s stl_scan system table to get dependent objects(tables).

```warehouse=# create table banandaram.test_lineage_for_ctas as select * from banandaram.lineage_dummy;
SELECT
Time: 8349.387 ms
warehouse=# select * from banandaram.test_lineage_for_ctas;
 foo
-----
   1
(1 row)

Time: 384.175 ms
warehouse=# select pg_last_query_id();
 pg_last_query_id
------------------
         55451607
(1 row)

Time: 590.522 ms
warehouse=# select tbl, perm_table_name from stl_scan where query = 55451607 and perm_table_name &lt;&gt; 'Internal Worktable' group by 1,2;
   tbl   |                                                             perm_table_name
---------+------------------------------------------------------------------------------------------------------------------------------------------
 8073220 | test_lineage_for_ctas
(1 row)

Time: 18629.544 ms```

just construct the urn and construct lineage mce by calling `builder.make_lineage_mce` with source and target urns and then emit metadata by calling `emitter.emit_mce`

you can use the redshift recipe to capture the rest of the metadata.

Screen Shot 2021-11-19 at 1.22.55 PM.png

It’s a little tricky for late binding views, but happy to help.

Hey <@U028SHHU3B4>, thank you very much for your swift response! Let me digest what you’ve said. Besides that, I strongly encourage you to create a PR for this part :slightly_smiling_face:

<@U028SHHU3B4> I'm happy to help to add this to Datahub. Do you mind if I ping you sometime? I tried out your example and I would like to ask some question about it.