Hi folks, is field level lineage supported at the moment? I see there was an RFC for it, and there is code in the repo corresponding to the RFCs, though the docs indicate it's still coming soon. https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage

The models exist for Dataset fields. We need to do a bit more work for retrieving them. You can theoretically add emitters for column level lineage similar to dataset lineage but we want to go a bit further than that and support SQL parsing as well. From a timeline standpoint, we will make progress on the backend side in May & June. You can expect support to land in early-mid July.

Good to know thanks. At the moment we've got an in-house orchestration system which we're considering replacing with Airflow though its not been confirmed yet. It hooks in with another in-house system we have for managing data pipelines within our data lake, that system has runs Juypter notebooks to do data transformations. I'm thinking of the following: • Airflow -> Use Airflow backend to declare dataset level lineage

If the transformation is being done through custom Python code and not standardized SQL (which is where parsing comes in), I agree that it would be brittle to decouple declaration of input/output columns from transformation logic.

Agree that it certainly makes sense to keep the lineage declarations coupled with the actual transformations to whatever extent possible, and then getting that information into the airflow lineage backend or emitting directly to datahub is a matter building out the plumbing. SQL parsing is the nice when possible, but unfortunately sql dialects are incredibly complex and data transformations are often done with a variety of tools.

I'll get back to you once I get a chance to look at our use case a bit more closely. Appreciate the input though

It is only dataset level lineage and not field-level

That's fine. Just to clarify, do you plan to allow for declaration of field level lineage when it can't be inferred automatically? I realise it will be a bit brittle potentially, though we could live with it.

Yes we will support emission of field level lineage

Hi folks, bringing this lineage thread back to life. Is column level lineage still in the plan for June/July release? Regardless of whether the answer is yes or no, do you have a mock-up/screenshot of what you would expect that lineage to look like on the user interface? Reason I ask is because we're evaluating various catalogs that would cover column level lineage; it would help to have an indication of what the functionality would look like on Datahub. We're leaning away from defining the lineage within Airflow, and going back to @gray-shoe-75895 suggestion of having it closer to our code. We'd likely build a small library to send those lineage definitions over to Datahub Another use case we have is allowing users to define their lineage on the UI, as opposed to through code. @loud-island-88694, this ties back to our discussion on Zoom. We have cases where we'd like to define datasets manually, we'd like to be able to define our lineage manually as well. Is this something you anticipate doing in the UI? Failing that, we could possibly build our own thin UI, and pass in the lineage from that to GMS as MCEs. @future-airplane-75730 @miniature-ram-76637

We are still planning to add the models and emitters in July timeframe

thanks very much for the update

Will do @loud-island-88694 👍