https://datahubproject.io logo
Join Slack
Powered by
Maybe <@U03BEML16LB> could help here... I deleted ...
# troubleshoot
n
Maybe @bulky-soccer-26729 could help here... I deleted one of the Glossary Nodes urn using the CLI but after that I am facing issues where frontend is not completely loading and I get the below error in GMS. Tagged Chris as I thought he was the one working closely with Glossary. Error stack in 🧵
Copy code
21:01:55.185 [pool-12-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Aarun.vasudevan) - batchGet - 200 - 6ms
21:01:55.201 [I/O dispatcher 1] INFO  c.l.m.k.e.ElasticsearchConnector:41 - Successfully feeded bulk request. Number of events: 1 Took time ms: -1
21:01:58.619 [ForkJoinPool.commonPool-worker-4] ERROR c.l.datahub.graphql.GmsGraphQLEngine:1396 - Failed to load Entities of type: GlossaryNode, keys: [urn:li:glossaryNode:d48c795c-5f3c-411f-8e4b-0e5a8a32522e, urn:li:glossaryNode:Classification] Failed to batch load GlossaryNodes
21:01:58.623 [ForkJoinPool.commonPool-worker-4] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to retrieve entities of type GlossaryNode
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
	at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.RuntimeException: Failed to retrieve entities of type GlossaryNode
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$null$167(GmsGraphQLEngine.java:1397)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	... 5 common frames omitted
Caused by: java.lang.RuntimeException: Failed to batch load GlossaryNodes
	at com.linkedin.datahub.graphql.types.glossary.GlossaryNodeType.batchLoad(GlossaryNodeType.java:79)
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$null$167(GmsGraphQLEngine.java:1394)
	... 6 common frames omitted
Caused by: java.lang.NullPointerException: null
	at com.linkedin.datahub.graphql.types.glossary.mappers.GlossaryNodeMapper.mapGlossaryNodeKey(GlossaryNodeMapper.java:57)
	at com.linkedin.datahub.graphql.types.common.mappers.util.MappingHelper.mapToResult(MappingHelper.java:22)
	at com.linkedin.datahub.graphql.types.glossary.mappers.GlossaryNodeMapper.apply(GlossaryNodeMapper.java:38)
	at com.linkedin.datahub.graphql.types.glossary.mappers.GlossaryNodeMapper.map(GlossaryNodeMapper.java:25)
	at com.linkedin.datahub.graphql.types.glossary.GlossaryNodeType.lambda$batchLoad$0(GlossaryNodeType.java:75)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at com.linkedin.datahub.graphql.types.glossary.GlossaryNodeType.batchLoad(GlossaryNodeType.java:77)
	... 7 common frames omitted
I don't see this urn in the metadata store
b
okay gotcha.. did the node that you deleted have any children?
in the UI, we don't let you delete Nodes that have children as this causes reference issues. A followup item that we want to get to is to cascade delete children but haven't been able to do that
have you done a hard delete or a soft delete? (hard delete requires a 
--hard
at the end)
n
I did a hard delete...
b
With which version of the CLI?
Were you asked this question?
Copy code
"Do you want to delete these references?"
n
No..I wasn't
b
And which version of the CLI?
Starting in 0.8.35 we have the delete references capability - which auto cleans up pointers to an entity on delete
n
Copy code
datahub@datahub-cli:/$ datahub version
DataHub CLI version:
Python version: 3.9.13 (main, May 28 2022, 13:56:03)
[GCC 10.2.1 20210110]
b
what do you get if you do 
datahub --version
?
oh woah
it's not printing your version?
n
It doesn't display the version
But, the frontend is on v0.8.38
Also the same with GMS
b
yeah it's possible to have different versions between those two
i'm thinking that you have an older CLI version that didn't use this referential delete functionality
b
CLI version should look something like this
Copy code
johnjoyce@Johns-MacBook-Pro-3 datahub-1 % datahub --version
acryl-datahub, version 0.8.20.0
are you working in a virtual environment?
n
Yes, I see I didn't upgrade my CLI pod for a while now...
b
Ah
That could be it
n
Missed when upgrading it
b
gotcha gotcha
i
@nutritious-bird-77396 Can you please update the cli pod to match your GMS version and re-run the delete command?
datahub delete --urn <urn> --hard
you should get the remove references prompt if any exist even if the entity no longer exists.
n
Got it..Thanks!
I am on the community image 
v0.8.38
Copy code
datahub@datahub-cli:/$ datahub --version
acryl-datahub, version 0.8.38+docker
datahub@datahub-cli:/$ datahub delete --urn "urn:li:glossaryNode:d48c795c-5f3c-411f-8e4b-0e5a8a32522e" --hard
This will permanently delete data from DataHub. Do you want to continue? [y/N]: y
[2022-06-30 00:04:21,559] INFO     {datahub.cli.delete_cli:134} - DataHub configured with <http://datahub-gms-service.data-platform.svc.cluster.local:8080>
Successfully deleted urn:li:glossaryNode:d48c795c-5f3c-411f-8e4b-0e5a8a32522e. 2 rows deleted
Took 3.332 seconds to hard delete 2 rows for 1 entities
Even after deleting the above urn thru the updated image I still get the same error in GMS.
I didn't see this message 
Do you want to delete these references?
Image used 
linkedin/datahub-ingestion:v0.8.38
b
cc @incalculable-ocean-74010 Can you take a look here?
i
Arun, please let me know when you're free to look into this more closely.
n
Sure, are you free after 12 CST? (I am not sure which zone you are in)
i
I’m on GMT +1. I’m free at 2pm CST (if my math is right)
n
Hey @incalculable-ocean-74010 I have a meeting from 2-3 would 3 CST work for you? Also, I am free between 12-2 CST as well.
i
Are you free now Arun?
n
@incalculable-ocean-74010 Yes I am free now
Just to keep everyone in sync after upgrading datahub version to 0.8.39 the issue was resolved.
b
glad to hear it 👍
2 Views
Open in Slack
PreviousNext
Powered by