https://datahubproject.io logo
Join Slack
Powered by
was wondering if anyone could shed some light on t...
# ingestion
b
was wondering if anyone could shed some light on this. im trying to programmatically create datasets using the example in /datahub/metadata-ingestion/examples/library/lineage_emitter_rest.py instead of using the CLI method. i could create institutional memory, dataset properties of a new dataset by modifying the example of lineage, and the dataset could be seen in the browser, not differently compared to the quickstart datasets. But there seem to be an extra step that I missed when i try creating a SchemaMetadataClass. the REST API accepted my request, but i cant browse the newly created dataset. the error message in the browser read:
Copy code
Exception while fetching data (/browse) : java.lang.RuntimeException: Failed to execute browse: entity type DATASET, path [prod, goonrtpe], filters: null, start: 0, count: 10
mae-consumer logs says:
Copy code
datahub-mae-consumer      | org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.linkedin.metadata.kafka.DataHubUsageEventsProcessor.consume(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.String, java.lang.String>)' threw exception; nested exception is java.lang.ClassCastException: com.linkedin.metadata.key.CorpUserKey cannot be cast to com.linkedin.identity.CorpUserInfo; nested exception is java.lang.ClassCastException: com.linkedin.metadata.key.CorpUserKey cannot be cast to com.linkedin.identity.CorpUserInfo
i've compared the data stored in MySQL for the datasets created using programmatically generated ones and the pipeline created ones and dont see a difference. I'm using a slightly older version of datahub v0.8.1
m
@better-orange-49102 I think there is a known bug in 0.8.1 related to product analytics, which shows up in this way. Could you upgrade to 0.8.3 and see if that fixes your issue?
b
i checked out v0.8.3 and erased the existing docker volume. the same error message still occurs after i ingested SchemaMetadata.
Copy code
Exception while fetching data (/browse) : java.lang.RuntimeException: Failed to execute browse: entity type DATASET, path [prod, myselfcreatedplatform], filters: null, start: 0, count: 10
The browse paths look ok from the MySQL end. though i did not see the 
com.linkedin.metadata.key.CorpUserKey cannot be cast to com.linkedin.identity.CorpUserInfo
error anymore
the underlying error is now : 
datahub-frontend-react    | 02:09:58 [ForkJoinPool.commonPool-worker-6] WARN  n.g.e.SimpleDataFetcherExceptionHandler - Exception while fetching data (/browse) : java.lang.RuntimeException: Failed to execute browse: entity type DATASET, path [prod, myselfcreatedplatform], filters: null, start: 0, count: 10
datahub-frontend-react | java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to execute browse: entity type DATASET, path [prod, myselfcreatedplatform], filters: null, start: 0, count: 10 datahub-frontend-react | at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) datahub-frontend-react | at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) datahub-frontend-react | at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592) datahub-frontend-react | at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) datahub-frontend-react | at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) datahub-frontend-react | at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) datahub-frontend-react | at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) datahub-frontend-react | at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) datahub-frontend-react | Caused by: java.lang.RuntimeException: Failed to execute browse: entity type DATASET, path [prod, myselfcreatedplatform], filters: null, start: 0, count: 10 datahub-frontend-react | at com.linkedin.datahub.graphql.resolvers.browse.BrowseResolver.lambda$get$1(BrowseResolver.java:50) datahub-frontend-react | at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) datahub-frontend-react | ... 5 common frames omitted datahub-frontend-react | Caused by: java.lang.RuntimeException: Failed to batch load Datasets datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:102) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.DatasetType.browse(DatasetType.java:143) datahub-frontend-react | at com.linkedin.datahub.graphql.resolvers.browse.BrowseResolver.lambda$get$1(BrowseResolver.java:41) datahub-frontend-react | ... 6 common frames omitted datahub-frontend-react | Caused by: com.linkedin.data.template.TemplateOutputCastException: Invalid URN syntax: Chars in namespace must be [a-z0-9-]!. Urn: urnli.dataPlatformmyselfcreatedplatform: urnli.dataPlatformmyselfcreatedplatform datahub-frontend-react | at com.linkedin.common.urn.DataPlatformUrn$1.coerceOutput(DataPlatformUrn.java:62) datahub-frontend-react | at com.linkedin.common.urn.DataPlatformUrn$1.coerceOutput(DataPlatformUrn.java:53) datahub-frontend-react | at com.linkedin.data.template.DataTemplateUtil.coerceOutput(DataTemplateUtil.java:954) datahub-frontend-react | at com.linkedin.data.template.RecordTemplate.obtainCustomType(RecordTemplate.java:365) datahub-frontend-react | at com.linkedin.schema.SchemaMetadata.getPlatform(SchemaMetadata.java:165) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.mappers.SchemaMapper.apply(SchemaMapper.java:25) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.mappers.SchemaMapper.map(SchemaMapper.java:15) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetSnapshotMapper.lambda$apply$0(DatasetSnapshotMapper.java:77) datahub-frontend-react | at java.util.ArrayList.forEach(ArrayList.java:1257) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetSnapshotMapper.apply(DatasetSnapshotMapper.java:55) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetSnapshotMapper.map(DatasetSnapshotMapper.java:40) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.DatasetType.lambda$batchLoad$0(DatasetType.java:96) datahub-frontend-react | at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) datahub-frontend-react | at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) datahub-frontend-react | at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) datahub-frontend-react | at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) datahub-frontend-react | at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) datahub-frontend-react | at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) datahub-frontend-react | at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) datahub-frontend-react | at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:100) datahub-frontend-react | ... 8 common frames omitted datahub-frontend-react | Caused by: java.net.URISyntaxException: Chars in namespace must be [a-z0-9-]!. Urn: urnli.dataPlatformmyselfcreatedplatform: urnli.dataPlatformmyselfcreatedplatform datahub-frontend-react | at com.linkedin.common.urn.Urn.validateAndExtractNamespace(Urn.java:380) datahub-frontend-react | at com.linkedin.common.urn.Urn.<init>(Urn.java:87) datahub-frontend-react | at com.linkedin.common.urn.Urn.createFromString(Urn.java:231) datahub-frontend-react | at com.linkedin.common.urn.DataPlatformUrn.createFromString(DataPlatformUrn.java:26) datahub-frontend-react | at com.linkedin.common.urn.DataPlatformUrn$1.coerceOutput(DataPlatformUrn.java:60) datahub-frontend-react | ... 27 common frames omitted
m
Seems like a typo @better-orange-49102 : urnlidataPlatform you have a . like urn:li.dataPlatform
b
yes! i need a new pair of glasses omg.
😄 1
thanks!
b
glad you got it working!
7 Views
Open in Slack
PreviousNext
Powered by