hi, is it possible to create lineage MCEs between ...
# ingestion
b
hi, is it possible to create lineage MCEs between ML models and Datasets/FeatureTables ? looking at this code in the Python lib:
Copy code
def make_lineage_mce(
    upstream_urns: List[str],
    downstream_urn: str,
    lineage_type: str = DatasetLineageTypeClass.TRANSFORMED,
) -> MetadataChangeEventClass:
    mce = MetadataChangeEventClass(
        proposedSnapshot=DatasetSnapshotClass(
            urn=downstream_urn,
            aspects=[
                UpstreamLineageClass(
                    upstreams=[
                        UpstreamClass(
                            dataset=upstream_urn,
                            type=lineage_type,
                        )
                        for upstream_urn in upstream_urns
                    ]
                )
            ],
        )
    )
    return mce
it looks like a
DatasetSnapshotClass
is required when defining the downstream URN, while I would. like to be able to set that as an ML Model or even FeatureTable
g
Hey @boundless-room-44377 - this request makes complete sense. Right now, the way you would make this connection would be via the MlFeature entity in MLFeatureProperties: https://github.com/linkedin/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/ml/metadata/MLFeatureProperties.pdl#L39
this lets you mark the inputs to a feature.
the mlfeatures that are inputs into the model
b
ah, that is super interesting
thanks @green-football-43791 we will try that out
g
great 🙂 let me know how it goes. We also have a demo of all of this here:

https://www.youtube.com/watch?v=rZsiB8z5rG4&t=2508s

The ML part of the townhall starts at 41:30
👍 2
👀 2