Hi everyone..
How do we resolve "Exception while f...
# uiw
worried-traffic-82020
08/19/2021, 4:15 AMHi everyone..
How do we resolve "Exception while fetching data (/browse/entities) : java.lang.RuntimeException: Failed to retrieve entities of type Dataset" in the UI ?
worried-traffic-82020
08/19/2021, 4:19 AMsorry i'm new to datahub..
worried-traffic-82020
08/19/2021, 4:19 AMdatahub is running in k8s
worried-traffic-82020
08/19/2021, 4:36 AMb
better-orange-49102
08/19/2021, 4:42 AMAh, same issue for me. I previously tried restarting gms and frontend and it worked for me one time. I'm running older version of datahub at the moment though. (Like 0.7x)
e
early-lamp-41924
08/19/2021, 4:52 AMCan you check the datahub-gms container logs?
Copy code
docker logs datahub-gmsb
better-orange-49102
08/19/2021, 6:03 AMgms logs has nothing that screams about exceptions. like Jim, im running on k8s (not using Helm chart, but each container is a deployment)
better-orange-49102
08/19/2021, 6:14 AMthis happens after i ingested a dataset and wanted to see it show up in the UI. It says
Copy code
SimpleDataFetcherExceptionHandler - Exception while fetching data (/browse) : java lang runtime exception: failed to execute browse: entity type DATASET, path [csv], filters: null, start: 0, count: 10 ...
caused by java.lang.RuntimeException: Failed to execute browse: entity type DATASET, path [csv], filters: null, start: 0, count: 10 ...
caused by : java.lang.RuntimeException: Failed to batch load Datasets ...
caused by com.linkedin.data.template.TemplateOutputCastException: Output has type java.lang.String, but does not have a registered coercer andd cannot be coerced to type java.net.URI....better-orange-49102
08/19/2021, 6:17 AMrebooting gms and frontend doesnt help. any ideas?
better-orange-49102
08/19/2021, 6:27 AMi noticed another similar thread taking place here, which suggest that ingestion is the culprit here..
https://datahubspace.slack.com/archives/CUMUWQU66/p1629351883371800
e
early-lamp-41924
08/19/2021, 6:57 AMWe will take a look at this asap. cc @green-football-43791 @big-carpet-38439
b
better-orange-49102
08/19/2021, 10:25 AMi have 2 instances running in k8s, and not using the datahub ingest libraries - rather, i build the MCEs and use the rest emitter to send the data to GMS. It ran ok initially (i could create the dataset and see it in the UI), then a few datasets later, the UI threw the error when i tried to navigate to the dataset.
b
big-carpet-38439
08/19/2021, 3:49 PMThis seems to be a case of a bad URL somewhere. You can see it's trying to cast a string to a URI... The newer versions should have better ingest-time validation to prevent this sort of data from getting in
big-carpet-38439
08/19/2021, 3:56 PM@better-orange-49102: We recently introduced more strict validations on ingest. I'd highly recommend bumping your version to get this. It will prevent some of these pieces of data getting into your system in the first place, and thus should help mitigate these read-time exceptions
big-carpet-38439
08/19/2021, 3:56 PM@worried-traffic-82020: Which versions of DataHub GMS container and ingestion framework are you working with?
b
better-orange-49102
08/20/2021, 12:47 AMAlright thanks for the explanation I'll upgrade the code and try it
better-orange-49102
08/20/2021, 1:11 AMAlso, I'll try to load the urls for each dataset individually and see which dataset has issues loading.
w
worried-traffic-82020
08/20/2021, 1:29 AM@big-carpet-38439 I was using the chart version 0.8.8. This issue happens after ingesting a lot of data, a roll back didn't seem to help as well.
worried-traffic-82020
08/20/2021, 1:30 AMI upgraded to 0.8.10 today and the issue seems to be resolved now. Didn't try a bulk load though..
b
better-orange-49102
08/20/2021, 11:04 AM@big-carpet-38439 for the sake of troubleshooting what part of the metadata causes rendering issues, can i check, does the UI directly load from mysql or ES when displaying a dataset? i want to fiddle with the string in mysql and see if it can resolve the display error, but am not sure if UI also queries ES when displaying the dataset.
This is because i can load some dataset directly in the UI, but not others. Am suspecting that those which cannot load are the ones causing the errors.
better-orange-49102
08/20/2021, 11:36 AMnvm, seems like an illegal string sequence causing the error
better-orange-49102
08/20/2021, 11:36 AMin the fieldpath description
better-orange-49102
08/20/2021, 11:48 AMremoving the substring allows the dataset to be rendered correctly, but the browsing hiearchy remains broken
e
early-lamp-41924
08/20/2021, 4:16 PM@better-orange-49102 What was the substring that was causing it?
b
big-carpet-38439
08/20/2021, 4:23 PMI'm surprised the browsing hierarchy is broken. Maybe we can all schedule a time to hop on in a huddle to see whats going on?
b
better-orange-49102
08/21/2021, 10:30 AMInside mysql, for the the description of a fieldPath, the string was being recorded as
"This is a "1_" string"better-orange-49102
08/21/2021, 10:32 AMI'll love to schedule a call, but because the instance is inside a intranet, it's hard to discuss also. I'll continue poking away and share any findings.
better-orange-49102
08/23/2021, 5:02 AMso if i delete away all the datasets that cannot be rendered in UI inside mySQL, the browsing and listing of datasets is restored (abeit i now have a bunch of datasets in the listings that are deleted). i assume this is because dataset listings uses information from both ES and mySQL.
b
big-carpet-38439
08/23/2021, 7:32 PMthis is correct @better-orange-49102
big-carpet-38439
08/23/2021, 7:32 PMIt seems that the string above needed to escaped. Do you know where this metadata was written from?
b
better-orange-49102
08/23/2021, 11:53 PMIt was a manually constructed MCE. We created a web form to allow users to create tables from flat file sources.
b
big-carpet-38439
08/24/2021, 12:17 AMI see - on the recent versions we will be doing stricter validation at write time, which should help prevent this type of data
4 Views