https://datahubproject.io logo
Join Slack
Powered by
Hello, team. Is it possible to ingest Data Freshne...
# ingestion
h
Hello, team. Is it possible to ingest Data Freshness via mcp with datasetUsageStatistics aspect? I can't see the corresponding attribute in the python class, though updated acryl-datahub to version 0.8.23.0
s
which python class is missing what?
h
I assumed that DatasetUsageStatisticsClass or DatasetProfileClass should have the new field to display Data Freshness, as in the demo it is displayed in the Stats tab of the dataset
Is there any information on how to ingest Data Freshness manually?
s
Can you please share with me the video and the UI label that you are referring to? I seem to be missing something here
h
s
Seems like this is controlled by a new aspect called operation which @echoing-airport-49548 recently added. I have not seen any examples separately but this https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/usage/snowflake_usage.py#L404 is how it is being done in snowflake usage. You can see the PR https://github.com/linkedin/datahub/pull/3845 for more details of all that was added in
h
Thank you, will try the OperationClass
Guess it's not that simple. Getting error 
'java.lang.RuntimeException: Unknown aspect operation for entity dataset'
trying to emit mcp like in the code provided
s
your GMS server is of what version?
h
I've pulled the latest version an hour ago and ran docker-compose, guess it should be latest? Any command to see it's version?
s
git pull or docker pull? If docker pull 
latest
tag try 
head
tag instead. Or better use 
v0.8.23
so we know the exact version being used
h
I used git pull
It says 
datahub-gms:head
Ok, I see now. Previously I was always using command 
datahub docker quickstart
and it pulled the latest dockers. This time I switched to using docker-compose directly (to use LDAP-authentication) and it did not pull the latest dockers. Now I got the latest version for sure. Ingestion finishes without errors, but when I try to open dataset in the UI I get a "SERVER_ERROR" with DataFetchingException for path["dataset", "operations"]
s
Can you check the server GMS for any errors?
h
Copy code
12:39:44.565 [ForkJoinPool.commonPool-worker-84] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: java.lang.NullPointerException
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
        at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.NullPointerException: null
        at com.linkedin.datahub.graphql.types.common.mappers.OperationMapper.apply(OperationMapper.java:34)
        at com.linkedin.datahub.graphql.types.common.mappers.OperationMapper.map(OperationMapper.java:17)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
        at com.linkedin.datahub.graphql.resolvers.load.TimeSeriesAspectResolver.lambda$get$0(TimeSeriesAspectResolver.java:65)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        ... 5 common frames omitted
Also this:
Copy code
12:39:44.621 [ForkJoinPool.commonPool-worker-155] ERROR c.datahub.graphql.GraphQLController:94 - Errors while executing graphQL query: "query getDataset($urn: String!) {\n  dataset(urn: $urn) 
...
errors: [DataHubGraphQLError{path=[dataset, operations], code=SERVER_ERROR, locations=[SourceLocation{line=102, column=5}]}
There is a lot of data in between, it doesn't seem to be useful
b
Thank you Sergey- we will have a fix out for this soon
cc @echoing-airport-49548
e
Hey Sergey, this should be fixed with this commit
Please let us know if you run into any other issues!
h
It works! Thanks a lot 🙌
e
Amazing, glad to hear it!
Open in Slack
PreviousNext
Powered by