https://datahubproject.io logo
Join Slack
Powered by
Is there any documentation showing how to create a...
# ingestion
c
Is there any documentation showing how to create an 
MLModel
entity using the java (or python) emitter?
c
Unfortunately, mlmodel creation example is not available readily. Can you please refer other examples here and try to create it? Basic concept is same for creating all entities in datahub. If not let me know.
c
Hi @careful-pilot-86309 unfortunately I am not able to get it, my code is below. Can you please advise?
Copy code
import com.linkedin.common.FabricType;
import com.linkedin.common.urn.DataPlatformUrn;
import com.linkedin.common.urn.MLModelUrn;
import com.linkedin.metadata.key.MLModelKey;
import com.linkedin.ml.metadata.MLModelProperties;
import datahub.event.MetadataChangeProposalWrapper;
import datahub.client.rest.RestEmitter;
import java.io.IOException;
import java.text.MessageFormat;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.ExecutionException;

public class MyModelAppTwo {
    public static void main(String[] args) throws IOException, ExecutionException, InterruptedException {

        MLModelUrn mlModelUrn = new MLModelUrn(new DataPlatformUrn("science"), "testModel", FabricType.PROD);

        MLModelKey mlModelKey = new MLModelKey()
                .setName("testModel")
                .setOrigin(FabricType.DEV)
                .setPlatform(new DataPlatformUrn("science"));

        MetadataChangeProposalWrapper mcpwOne = MetadataChangeProposalWrapper.builder()
                .entityType("mlModel")
                .entityUrn(mlModelUrn)
                .upsert()
                .aspect(mlModelKey)
                .aspectName("mlModelKey")
                .build();

        MLModelProperties mlModelProperties = new MLModelProperties()
                .setDescription("some desc");

        MetadataChangeProposalWrapper mcpwTwo = MetadataChangeProposalWrapper.builder()
                .entityType("mlModel")
                .entityUrn(mlModelUrn)
                .upsert()
                .aspect(mlModelProperties)
                .aspectName("mlModelProperties")
                .build();

        /** Emit **/
        RestEmitter emitter = RestEmitter.create(b -> b.server("<http://localhost:8080>"));

        List<MetadataChangeProposalWrapper> mcpws = Arrays.asList(
                mcpwOne,
                mcpwTwo
        );

        for (MetadataChangeProposalWrapper mcpw : mcpws) {
            System.out.println(MessageFormat.format("Starting metadata emission for {0} ...", mcpw));
            emitter.emit(mcpw, null).get();
        }

        System.exit(0);
    }
}
^ note: the code itself runs successfully, but no entities are visible on the datahub UI
Note: I discovered that it's actually a problem with the UI; the entity gets to datahub, but the MLModel entities do not appear when clicking on "ML Models" in the UI (though they can be found by searching for them) I am using the datahub quickstart locally with the newest docker images
b
@colossal-sandwich-50049 This is because there has been no Browse Paths produced for the ML model. You can try to produce the "browsePaths" aspect for the ML Model Entity, which will enable it to show in the UI 🙂
Let me try to find an exampe
Copy code
browse_path = BrowsePathsClass(
            paths=["/powerbi/{}".format(self.__config.workspace_id)]
        )
        return MetadataChangeProposalWrapper(
            entityType="dataset",
            changeType="UPSERT",
            entityUrn="urn:li:dataset:(urn:li:dataPlatform:custom,MyFileName,PROD),
            aspectName="browsePaths",
            aspect=browse_path,
        )
(Except you'd set the entitytype as mlModel, of course
c
Here is the exact example in java:
Copy code
BrowsePaths browse_path = new BrowsePaths().setPaths(new StringArray("/mlModels/test"));
        MetadataChangeProposalWrapper mcpw = MetadataChangeProposalWrapper.builder()
            .entityType("mlModel")
            .entityUrn(mlModelUrn)
            .upsert()
            .aspect(browse_path)
            .aspectName("browsePaths")
            .build();
Please note that browse path must start with "/mlModels/"
c
Thank you both!
b
I don't believe it has to
But that will put it in the ml models directory
Open in Slack
PreviousNext
Powered by