https://datahubproject.io logo
Join Slack
Powered by
Hi All. Question about `Historical roadmap` - part...
# getting-started
a
Hi All. Question about 
Historical roadmap
- part 
No-code Metadata Model Additions
- I see that 
No need to write any code (in GraphQL or UI) to visualize metadata
is not checked while stay in 
Historical part.
Does that mean 
abandoned
, or just delayed?
b
Hi @ambitious-airline-8020! We have not completely removed it, but we are awaiting signal from the community that they would benefit from it, because it's a nontrivial amount of additional effort on the part of the developer core. Is this a feature your company would benefit from? If you wouldn't mind, I'd love to better understand your use case 🙂
a
Well, I am not the one who is able to talk from the Company 😉 But as for use case. I got 2 issues while tried to implement custom entity, having 
@Relationship
to Dataset entity: 1. UI is broken after you open linked Dataset entity and click 
Lineage
tab (browser error, I believe related to that fact, that I have just onboarded PDL, not touhed any React/Typescript things). I suppose no-code UI aimed to resolve it. 2. GraphQL querying makes a lot of interest. But at this moment it makes some pain to deal with that mappers to introduce custom entity
b
Okay so #1 should definitely not happen. This must be a bug. cc. @green-football-43791 about Lineage resolver.. #2 I agree it does take some pain. The question is whether the pain x the frequency of the operation is worth the investment of time engineering auto graphql generation. It might be, we will certainly add your input to our discussions on the topic. Out of curiosity, what is the entity type you are hoping to add?
g
Hey @ambitious-airline-8020 - could you share what change you made to the Dataset entity specifically? Adding a @Relationship pointing to an existing entity should be no problem. If you tied a relationship to a new Entity, you'll need to update the UrnToEntityMapper
a
@green-football-43791 I did update UrnToEntityMapper as well
Let me share what I did
Error was only on Chrom dev tools console, I do not remember strictly, it was some render error. But it caused whoole page brake
Sharing just most important. Mapper
Copy code
Index: datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/common/mappers/UrnToEntityMapper.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/common/mappers/UrnToEntityMapper.java b/datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/common/mappers/UrnToEntityMapper.java
--- a/datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/common/mappers/UrnToEntityMapper.java	(revision a483933eab6d072ba6d2a414e378eee1e80dbdc6)
+++ b/datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/common/mappers/UrnToEntityMapper.java	(date 1623398038598)
@@ -10,6 +10,7 @@
 import com.linkedin.datahub.graphql.generated.Dataset;
 import com.linkedin.datahub.graphql.generated.Entity;
 import com.linkedin.datahub.graphql.generated.Tag;
+import com.linkedin.datahub.graphql.generated.Workspace;
 import com.linkedin.datahub.graphql.types.mappers.ModelMapper;
 import javax.annotation.Nonnull;
 
@@ -48,6 +49,10 @@
       partialEntity = new Tag();
       ((Tag) partialEntity).setUrn(input.toString());
     }
+    if (input.getEntityType().equals("workspace")) {
+      partialEntity = new Workspace();
+      ((Workspace) partialEntity).setUrn(input.toString());
+    }
     if (input.getEntityType().equals("corpuser")) {
       partialEntity = new CorpUser();
       ((CorpUser) partialEntity).setUrn(input.toString());
Index: datahub-graphql-core/src/main/java/com/datarobot/datahub/graphql/types/workspace/mappers/WorkspaceSnapshotMapper.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/datahub-graphql-core/src/main/java/com/datarobot/datahub/graphql/types/workspace/mappers/WorkspaceSnapshotMapper.java b/datahub-graphql-core/src/main/java/com/datarobot/datahub/graphql/types/workspace/mappers/WorkspaceSnapshotMapper.java new file mode 100644 --- /dev/null (date 1623317140667) +++ b/datahub-graphql-core/src/main/java/com/datarobot/datahub/graphql/types/workspace/mappers/WorkspaceSnapshotMapper.java (date 1623317140667) @@ -0,0 +1,45 @@ +package com.xxx.datahub.graphql.types.workspace.mappers; + +import com.xxx.metadata.snapshot.WorkspaceSnapshot; +import com.xxx.workspace.WorkspaceInfo; +import com.xxx.workspace.WorkspaceProperties; +import com.linkedin.datahub.graphql.generated.EntityType; +import com.linkedin.datahub.graphql.generated.Workspace; +import com.linkedin.datahub.graphql.types.common.mappers.StringMapMapper; +import com.linkedin.datahub.graphql.types.mappers.ModelMapper; +import com.linkedin.metadata.dao.utils.ModelUtils; + +import javax.annotation.Nonnull; + + +/** + * Maps Pegasus {@link RecordTemplate} objects to objects conforming to the GQL schema. + * + * To be replaced by auto-generated mappers implementations + */ +public class WorkspaceSnapshotMapper implements ModelMapper<WorkspaceSnapshot, Workspace> { + + public static final WorkspaceSnapshotMapper INSTANCE = new WorkspaceSnapshotMapper(); + + public static Workspace map(@Nonnull final WorkspaceSnapshot workspaceSnapshot) { + return INSTANCE.apply(workspaceSnapshot); + } + + @Override + public Workspace apply(@Nonnull final WorkspaceSnapshot workspaceSnapshot) { + final Workspace result = new Workspace(); + result.setUrn(workspaceSnapshot.getUrn().toString()); + result.setType(EntityType.WORKSPACE); + result.setName(workspaceSnapshot.getUrn().getEntityKey().get(0)); + + ModelUtils.getAspectsFromSnapshot(workspaceSnapshot).forEach(aspect -> { + if (aspect instanceof WorkspaceProperties) { + final WorkspaceProperties wsProperties = (WorkspaceProperties) aspect; + result.setCustomProperties(StringMapMapper.map(wsProperties.getCustomProperties())); + } else if (aspect instanceof WorkspaceInfo) { + result.setDescription(((WorkspaceInfo) aspect).getDescription()); + } + }); + return result; + } +}
Index: datahub-web-react/src/graphql/workspace.graphql IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/datahub-web-react/src/graphql/workspace.graphql b/datahub-web-react/src/graphql/workspace.graphql new file mode 100644 --- /dev/null (date 1623325521607) +++ b/datahub-web-react/src/graphql/workspace.graphql (date 1623325521607) @@ -0,0 +1,7 @@ +query getWorkspace($urn: String!) { + workspace(urn: $urn) { + urn + name + description + } +}
Index: datahub-web-react/datahub-frontend.graphql IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/datahub-web-react/datahub-frontend.graphql b/datahub-web-react/datahub-frontend.graphql --- a/datahub-web-react/datahub-frontend.graphql (revision a483933eab6d072ba6d2a414e378eee1e80dbdc6) +++ b/datahub-web-react/datahub-frontend.graphql (date 1623325414040) @@ -7,6 +7,7 @@ type Query { dataset(urn: String!): Dataset + workspace(urn: String!): Workspace user(urn: String!): CorpUser search(input: SearchInput!): SearchResults autoComplete(input: AutoCompleteInput!): AutoCompleteResults @@ -290,6 +291,7 @@ DATA_JOB CORP_USER CORP_GROUP + WORKSPACE } # Search Input @@ -381,3 +383,20 @@ type: EntityType! urn: String! } + +type Workspace { + urn: String! + name: String! + + """ + Description of the workspace + """ + description: String + + """ + Properties of the workspace + """ + customProperties: [PropertyTuple!] + + +} \ No newline at end of file
Index: metadata-models/src/main/pegasus/com/datarobot/workspace/WorkspaceInfo.pdl IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/metadata-models/src/main/pegasus/com/datarobot/workspace/WorkspaceInfo.pdl b/metadata-models/src/main/pegasus/com/datarobot/workspace/WorkspaceInfo.pdl new file mode 100644 --- /dev/null (date 1623073114293) +++ b/metadata-models/src/main/pegasus/com/datarobot/workspace/WorkspaceInfo.pdl (date 1623073114293) @@ -0,0 +1,38 @@ +namespace com.xxx.workspace + +import com.linkedin.common.Urn +import com.linkedin.common.DatasetUrn + +/** + * Info associated with a Workspace + */ +@Aspect = { + "name": "workspaceInfo" +} +record WorkspaceInfo { + + /** + * Description of the service + */ + @Searchable = {} + description: string + + /** + * The owners of the workspace + */ + @Relationship = { + "name": "OwnedBy" + } + owner: Urn + + /** + * Datasets in a workspace + */ + @Relationship = { + "/*": { + "name": "Contains", + "entityTypes": [ "dataset" ] + } + } + datasets: array[DatasetUrn] = [ ] +} \ No newline at end of file
GraphQL queries works fine, tested it with client, deployed graphql service I believe it is related with UI
let me know if you want to see screenshot for an error
b
Oh interesting. Maybe we can hop on a call to debug this further?
a
Hi @big-carpet-38439 This and next week I am traveling. But let me make some screencast at least. Will post it later today
Ok, so here the screen cast. I will split it into logical parts
PART 1: chrome error itself URL: 
<http://localhost:9002/dataset/urn:li:dataset:(urn:li:dataPlatform:hdfs,TrainingDataset,PROD)/lineage>
Context: Custome entity 
Workspace
, has 
@Relationship
field to Dataset (
TraingingDataset
). PDL - see above in a thread
No errors in docker output for containers (launched as 
./dev.sh
)
^^^ PART 2: showing dataset via CURL
Copy code
curl -H 'X-RestLi-Protocol-Version:2.0.0' -H 'X-RestLi-Method: get' '<http://localhost:8080/datasets/($params:(),name:TrainingDataset,origin:PROD,platform:urn%3Ali%3AdataPlatform%3Ahdfs)>' | jq
PART 3: get workspace, linked to a dataset
Copy code
curl '<http://localhost:8080/entities/urn%3Ali%3Aworkspace%3Amydemoworkspace>' -H 'X-RestLi-Protocol-Version:2.0.0' | jq
@big-carpet-38439 please feel free to request additional info - I will find a way to grab more between trips
g
Ah @ambitious-airline-8020 the error you are reporting is coming from here:
If you haven't already, you'll need to create an entity class in React as well that describes how the entity should be rendered
a
Hi @green-football-43791. Thanks - I will try to deal with App.tsx and let you know on feedback today
THanks @green-football-43791 - it works perfectly - changed App.tsx, implemented UI part
b
Awesome!!!
g
Terrific!
Open in Slack
PreviousNext
Powered by