https://datahubproject.io logo
Join Slack
Powered by
Hello all, started evaluating data hub for a proj...
# ingestion
h
Hello all, started evaluating data hub for a project I'm working on. What is the simplest way to just register a file located in s3 as a dataset? I'm using the rest api and I Tried a metadatchangeproposal for a dataset entity with datasetproperties class with a uri set. It submits fine but I get a 500 error from the web app.
m
are you running datahub locally?
h
Yep
m
if so, could you please check datahub-gms logs from docker when you visit the page?
h
Copy code
Seems to be two diff errors and here they are:ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler - Failed to execute DataFetcher
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to retrieve entities of type Dataset
       at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
       at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
       at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
       at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
       at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
       at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
       at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
       at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.RuntimeException: Failed to retrieve entities of type Dataset
       at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$null$114(GmsGraphQLEngine.java:854)
       at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
       ... 5 common frames omitted
Caused by: java.lang.RuntimeException: Failed to batch load Datasets
       at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:108)
       at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$null$114(GmsGraphQLEngine.java:851)
       ... 6 common frames omitted
Caused by: com.linkedin.data.template.TemplateOutputCastException: Output http:///data/tracking/PageViewEvent has type java.lang.String, but does not have a registered coercer and cannot be coerced to type java.net.URI
       at com.linkedin.data.template.DataTemplateUtil.coerceOutput(DataTemplateUtil.java:950)
       at com.linkedin.data.template.RecordTemplate.obtainCustomType(RecordTemplate.java:365)
       at com.linkedin.dataset.DatasetProperties.getUri(DatasetProperties.java:282)
       at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetSnapshotMapper.lambda$apply$0(DatasetSnapshotMapper.java:70)
       at java.util.ArrayList.forEach(ArrayList.java:1259)
       at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetSnapshotMapper.apply(DatasetSnapshotMapper.java:55)
       at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetSnapshotMapper.map(DatasetSnapshotMapper.java:40)
       at com.linkedin.datahub.graphql.types.dataset.DatasetType.lambda$batchLoad$0(DatasetType.java:102)
       at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
       at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
       at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
       at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
       at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
       at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
       at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:106)
       ... 7 common frames omitted
Copy code
{\n   time\n   __typename\n }\n __typename\n}\n\nfragment globalTagsFields on GlobalTags {\n tags {\n   tag {\n     urn\n     name\n     description\n     __typename\n   }\n   __typename\n }\n __typename\n}\n\nfragment glossaryTerms on GlossaryTerms {\n terms {\n   term {\n     urn\n     name\n     __typename\n   }\n   __typename\n }\n __typename\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=3, column=5}], path=[browse, entities], extensions={code=500, type=SERVER_ERROR, classification=DataFetchingException}}], data={browse=null}}, errors: [DataHubGraphQLError{path=[browse, entities], code=SERVER_ERROR, locations=[SourceLocation{line=3, column=5}]}]
Copy code
lineage_mcp = MetadataChangeProposalWrapper(
    entityType="dataset",
    changeType=ChangeTypeClass.UPSERT,
    entityUrn=builder.make_dataset_urn("bigquery", "downstream3"),
    aspectName="datasetProperties",
    aspect=DatasetPropertiesClass(
      uri="http:///data/tracking/PageViewEvent",
          description="test.description",
        )
  )
 
  # Create an emitter to the GMS REST API.
  emitter = DatahubRestEmitter("<http://localhost:8080>")
 
  # Emit metadata!
  emitter.emit_mcp(lineage_mcp)
and this is my python code for the request
m
Thanks @happy-father-45304. I am able to reproduce. Seems like the script is fine and the URI also looks okay even with removing an extra "/". So not sure what is going on but will take a look.
Hi @happy-father-45304 Can you please try externalUrl="...." field. Seems like the uri field should be deprecated.
h
that was it, seems to be working now. Thanks!
2 Views
Open in Slack
PreviousNext
Powered by