Hi everyone, Is there any way to ingest column level lineage to datahub?

Fine Grained Lineage - https://datahubproject.io/docs/generated/metamodel/entities/dataset/#fine-grained-lineage

What will be the best parser to find column level lineage out of sql queries?

I don't believe a great option (or an option sanctioned by us) is available cc @gray-shoe-75895 for any input!

Yep I’ve done a decent bit of research into this, and haven’t found anything that works well yet

Okay. As column level lineage is supported by snowflake in datahub, Is it a good way to use sql queries with snowflake and then ingest that data through snowflake to datahub? Will that work?

Snowflake has a system table which contains some of the query and access history, and we pull the column-level lineage from there. We’re not doing any sql parsing for snowflake queries, but we do get column-level lineage automatically. In other words, you’ll see column-level lineage for any queries issued against snowflake if you use the datahub snowflake ingestion source