A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi all!
I have a dataset (s3) and a table (snowflake) already ingested.
How can I indicate lineage between them (s3 is the upstream of / comes before snowflake)?

hey Ben! you can always manually specify lineage using file-based ingestion as outlined in our docs here: <https://datahubproject.io/docs/generated/ingestion/sources/file-based-lineage/#lineage-file-format>

What system do you use to load data from s3 - snowflake? 

<@UV0M2EB8Q>
We ingest from s3 to Snowflake using Snowpipe

<@U03BEML16LB>
This is the method I used, but unfortunately I've read that only dataset-to-dataset is supported.
I ended up with a new "dataset" of snowflake with identical name to the original, real snowflake table (that's problem #1).
Another problem is that the new snowflake dataset that was created does not seem to show any lineage relation to the s3 dataset

I'm sure there is a more automated way to extract this lineage if you are doing a standard load using snowpipe

<@UV0M2EB8Q> I was sure too, but specifically with s3 I only ran into what Chris suggested, or some python version of it. The plan is to integrate the pythonic way into the code that creates the Snowpipes.
But this still doesn't work as I would expect

If you are getting identical looking entities in snowflake, take a look at the urn that you have carefully for the two “duplicate” entities. There is probably a small difference there. 

You can copy the urn from the UI using the “copy” button on the top right on the entity page 

<@UV0M2EB8Q> thanks a lot! That's exactly what I was missing