A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi team, we are looking into ingest ML metadata to DataHub. Can i check the rationale why DataHub decided to model feature groups (`FeatureTable`) as first-class entity instead of features (`Features`), which was proposed in <https://datahubproject.io/docs/rfc/active/1812-ml_models|this RFC>? Without features as first-class entities, how could we enable dataset-to-feature and feature-to-model lineage?

The short answer is we probably over-fitted to the Sagemaker way of doing things. It completely makes sense to have Feature as a separate entity and also model the dataset-&gt;feature-&gt;feature table-&gt;model and also dataset-&gt;feature-&gt;model lineage

We also need a separate entity page for feature which we currently don't have and our lineage visualization also needs to improve to support what is effectively column-level lineage

What is your timeline for this and also do you have bandwidth for contributions?

Hey Swaroop, we are thinking of picking this up in our next quarter, starting in January – both a separate entity page for feature and some lineage work – very likely to start with dataset-&gt;feature-&gt;model.

In terms of contributions, don’t want to overpromise, but I believe the team will be eager to contribute and we’d be happy to prioritize some time for this

good to know. We can look into cleaning up the model in Q1 and collaborate with you

<@U025KHBRR3L> <@U01C3DMG2GL> Hi, I wanted to touch base and check the status of the <https://datahubproject.io/docs/rfc/active/1812-ml_models|proposed RFC>. Thank you!

I noticed that the PDLs  are inside, thanks for that! <https://github.com/linkedin/datahub/pull/1721/files>
but I also see it's not part of the entity-registry.yml: <https://github.com/linkedin/datahub/blob/master/metadata-models/src/main/resources/entity-registry.yml>, does it means I need to add them to the registry by myself?

Hey <@U033A89P962> - they were introduced into the metadata model graph via the Snapshot `metadata-models/src/main/pegasus/com/linkedin/metadata/snapshot/MLFeatureSnapshot.pdl`. This was the legacy way of introducing models into the metadata model graph, essentially entity-registry as code. The entity registry is the new way, but you will still be able to access these old models since they were introduced in the deprecated way.

Screen Shot 2022-03-02 at 2.51.05 PM.png

Hi <@U033A89P962> a convenient way to explore the model is in the <https://datahubproject.io/docs/|docs> sidebar -&gt; e.g. (<https://datahubproject.io/docs/generated/metamodel/entities/mlfeature>). This should give you an easier way to see what all exists and is modeled.

<@U01JUBAPZPS> Thanks for your response!, I have a question about the new way of defining entities, so we have this: <https://github.com/linkedin/datahub/blob/master/metadata-models/src/main/resources/entity-registry.yml>, but where can I find the actual definitions of the aspects for instance? is it going to be in the `PDLs` - is there a new way beside of the `PDLs` to put those definitions?