wooden-spring-14915
02/03/2022, 3:17 PMlimited-solstice-12595
happy-scooter-645
02/03/2022, 4:25 PMelegant-house-93198
happy-scooter-645
02/04/2022, 8:09 AMpush lineage information to the Secoda API, similar to OpenLineageHey Andrew, OpenLineage already seems to be becoming the standard for data lineage collection, and Marquez which is the leading implementation of the lineage repository offers out-of-the-box integration with popular tools (Spark, dbt, Airflow). That makes it very tempting for data teams to just use this toolset. I don't want to speak for my fellow Secoda customers, but I'd rather use the existing integrations that OpenLineage+Marquez offers than writing custom code to push metadata from e.g. Airflow to the Secoda API.
Alternatively, if there are many people who already use OpenLineage, we could treat it similar to dbt’s lineage info where we make an integration for it.So clearly, I am more interested in this approach. Even if not many people use OpenLineage yet, having an OpenLineage->Secoda integration can encourage many to look into it