Hi I am trying to use Java Emitter to create entit...
# ingestion
e
Hi I am trying to use Java Emitter to create entities of dataflow and datajobs linked to it. Those jobs will have relations to datasets as input and output. However when i tried to write dataflow:
Copy code
MetadataChangeProposalWrapper mcpw = MetadataChangeProposalWrapper.builder()
        .entityType("dataflow")
        .entityUrn("urn:li:dataflow:(urn:li:dataPlatform:kafka,trace-pipeline,PROD)")
        .upsert()
        .aspect(new DataFlowInfo()
                .setName("Trace pipeline")
                .setDescription("Pipeline for trace service")
        )
        .build();
i am able to successfully emit it
Copy code
emitter.emit(mcpw, new Callback()
but then when executing graphql query:
Copy code
graphql query
{
  search(input: { type: DATA_FLOW, query: "*", start: 0, count: 10 }) {
    start
    count
    total
    searchResults {
      entity{
        urn
        type
        ...on DataFlow {
            cluster
         }
      }
    }
  }
}
i get following error:
Copy code
response
{
  "errors": [
    {
      "message": "The field at path '/search/searchResults[0]/entity' was declared as a non null type, but the code involved in retrieving data has wrongly returned a null value.  The graphql specification requires that the parent field be set to null, or if that is non nullable that it bubble up null to its parent and so on. The non-nullable type is 'Entity' within parent type 'SearchResult'",
      "path": [
        "search",
        "searchResults",
        0,
        "entity"
      ],
      "extensions": {
        "classification": "NullValueInNonNullableField"
      }
    }
  ],
  "data": {
    "search": null
  }
}
so basically how such dataflow entity should look like? Did i miss some required fields? And how from entities documentation i can know which fields are optional and which are mandatory? thx
l
@green-football-43791 is this related to the graphql regression? ^
e
I am using version v0.8.23
I figured out what was my issue when i checked what returns graphql query available on demo. Afterwards i was able to modify my metadataChangeProposal and its working fine.
Copy code
private static void createDataPipeline() throws IOException, ExecutionException, InterruptedException {
    MetadataChangeProposalWrapper mcpw = MetadataChangeProposalWrapper.builder()
            .entityType("dataflow")
            .entityUrn("urn:li:dataFlow:(kafka,trace-pipeline,PROD)")
            .upsert()
            .aspect(new DataFlowInfo()
                    .setName("Trace pipeline")
                    .setDescription("Pipeline for trace service")
            )
            .build();
however i would like to highlight that its easy to break the ui since the validation of proposed entity is not sufficient....
l
@big-carpet-38439 ^
b
@orange-night-91387
Small URN issue is causing this to blow up. Another reason we need urn validation 🙂
plus1 2