Hey team I am testing my new entity type locally I successfu DataHub #troubleshoot

Hey team, I am testing my new entity type locally....

red-napkin-59945

03/02/2022, 1:01 AM

Hey team, I am testing my new entity type locally. I successfully write the entity record but failed to get the entity with error:

java.lang.UnsupportedOperationException: Failed to find Typeref schema associated with Config-based Entity

red-napkin-59945

03/02/2022, 1:01 AM

Here is the stack trace:

Copy code

Caused by: java.lang.UnsupportedOperationException: Failed to find Typeref schema associated with Config-based Entity
        at com.linkedin.metadata.models.ConfigEntitySpec.getAspectTyperefSchema(ConfigEntitySpec.java:78)
        at com.linkedin.metadata.entity.EntityService.toAspectUnion(EntityService.java:765)
        at com.linkedin.metadata.entity.EntityService.lambda$null$8(EntityService.java:632)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
        at com.linkedin.metadata.entity.EntityService.lambda$getLatestAspectUnions$9(EntityService.java:633)
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
        at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
        at com.linkedin.metadata.entity.EntityService.getLatestAspectUnions(EntityService.java:630)
        at com.linkedin.metadata.entity.EntityService.getSnapshotRecords(EntityService.java:619)
        at com.linkedin.metadata.entity.EntityService.getSnapshotUnions(EntityService.java:611)
        at com.linkedin.metadata.entity.EntityService.getEntities(EntityService.java:510)
        at com.linkedin.metadata.entity.EntityService.getEntity(EntityService.java:495)
        at com.linkedin.metadata.resources.entity.EntityResource.lambda$get$0(EntityResource.java:116)
        at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:30)
        ... 85 common frames omitted

red-napkin-59945

03/02/2022, 1:02 AM

Is the

Config-based Entity

not supported by Restli endpoint?

red-napkin-59945

03/02/2022, 1:03 AM

cc: @big-carpet-38439 (seems you are oncall 🙂 )

big-carpet-38439

03/02/2022, 1:25 AM

Oh interesting

big-carpet-38439

03/02/2022, 1:26 AM

Can you post the CURL you are making?

big-carpet-38439

03/02/2022, 1:26 AM

I've been testing this recently actually,... It just depends which read endpoint you are using (there are unfortunately a few)

red-napkin-59945

03/02/2022, 1:34 AM

Copy code

@Test
  public void test() throws Exception {
    DataDocUrn dataDocUrn = new DataDocUrn("querybook", "123456");
    MetadataChangeProposalWrapper wrapper = MetadataChangeProposalWrapper.builder()
        .entityType("dataDoc")
        .entityUrn(dataDocUrn.toString())
        .upsert()
        .aspect(new DataDocInfo()
            .setDataDocUrl(new Url("<https://querybook.com//datadoc/784>"))
            .setTitle("Test")
            .setDescription("test")
            .setLastModified(new ChangeAuditStamps()))
        .build();

    EventFormatter eventFormatter = new EventFormatter(EventFormatter.Format.PEGASUS_JSON);
    ObjectMapper objectMapper = new ObjectMapper().setSerializationInclusion(JsonInclude.Include.NON_NULL);
    JacksonDataTemplateCodec dataTemplateCodec = new JacksonDataTemplateCodec(objectMapper.getFactory());
    String eventString = dataTemplateCodec.mapToString(eventFormatter.convert(wrapper).data());
    RestEmitter emitter = RestEmitter.create(b -> b.server("<http://localhost:8080>"));
    MetadataWriteResponse writeResponse = emitter.emit(wrapper).get();
  }

red-napkin-59945

03/02/2022, 1:35 AM

I wrote a unit test to do this

red-napkin-59945

03/02/2022, 1:37 AM

I think the endpoint is `/aspects?action=ingestProposal`endpoint according to the url

<http://localhost:8080/aspects?action=ingestProposal>

red-napkin-59945

03/02/2022, 1:38 AM

The curl cmd to get the entity is

curl '<http://localhost:8080/entities/urn%3Ali%3AdataDoc%3A(querybook,123456)>'

big-carpet-38439

03/02/2022, 1:41 AM

Oh got it

big-carpet-38439

03/02/2022, 1:41 AM

So the part that will fail unfortunately is the entities CURL

big-carpet-38439

03/02/2022, 1:42 AM

We've recently moved to a "V2" of the entities endpoint which allows you to fetch for entities created by the Config Entity Registry. The legacy endpoint only supports deprecated snapshots 😕

big-carpet-38439

03/02/2022, 1:42 AM

Let me send the curl

red-napkin-59945

03/02/2022, 1:42 AM

ah,

red-napkin-59945

03/02/2022, 1:42 AM

Thanks for the upate

big-carpet-38439

03/02/2022, 1:42 AM

Copy code

curl '<http://localhost:8080/entitiesV2/><urn>'

big-carpet-38439

03/02/2022, 1:43 AM

Try that one and let's see what we get

red-napkin-59945

03/02/2022, 1:43 AM

curl '<http://localhost:8080/entitiesV2/urn%3Ali%3AdataDoc%3A(querybook,123456)>

this works

red-napkin-59945

03/02/2022, 1:45 AM

maybe we annotate the GET and BATCH_GET method as deprecated in

EntityResource.java

red-napkin-59945

03/02/2022, 1:45 AM

anyway, thanks for checking!!

big-carpet-38439

03/02/2022, 1:49 AM

amazing

big-carpet-38439

03/02/2022, 1:49 AM

Yeah we should definitely do that

big-carpet-38439

03/02/2022, 1:49 AM

We've already migrated our own internal callers to the v2, just need to publicly deprecate that one now

👍 1

red-napkin-59945

03/02/2022, 5:46 AM

Could you help review this curl cmd to see why the server complain could not parse the request?

Copy code

curl '<http://localhost:8080/aspects?action=ingestProposal>' -X POST --data '{
   "proposal":{
      "aspectName":"dataDocInfo",
      "entityUrn":"urn:li:dataDoc:(querybook,123456)",
      "entityType":"dataDoc",
      "changeType":"UPSERT",
      "aspect":{
         "contentType":"application/json",
         "value":"{
            \"description\":\"test\",
            \"lastModified\":{},
            \"title\":\"Test\",
            \"dataDocUrl\":\"<https://querybook.com/lasik/datadoc/784>\"
         }"
      }
   }
}'

hundreds-photographer-13496

03/02/2022, 6:41 AM

I am not an expert, but you may want to double check lastModified's value. If its of type AuditStamp, you will need to populate the values for for time and actor fields like this

big-carpet-38439

03/02/2022, 4:18 PM

Correct^ this is likely due to missing a field

red-napkin-59945

03/02/2022, 4:20 PM

but the value string is generated from test code, like

Copy code

String eventString = dataTemplateCodec.mapToString(eventFormatter.convert(wrapper).data());

red-napkin-59945

03/02/2022, 4:22 PM

let me try to fill the

lastModified

plus1 1

6 Views

Open in Slack

Previous Next