Hi all, After changing the key of the Dataset ent...
# troubleshoot
m
Hi all, After changing the key of the Dataset entity I am running into issues. After deleting all datasets, deploying the changes and inserting the datasets again with the new key format I get some errors on the home screen of the application. Even when I don't insert new datasets I get the same errors. The same happens for a custom entity we made, dataproduct:
Copy code
Caused by: java.lang.RuntimeException: Failed to batch load data products
	at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
	... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
	at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
	at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
	at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
	at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
	at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
	at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
	at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
	... 3 common frames omitted
After some looking around I found that there was some data in a hidden ElasticSearch index,
.ds-datahub_usage_event-000001
, that still seemed to contain rows with URNs that refer to data that was deleted. it looks like the recommendations/suggestions section on the main page uses this index and as such gets some data that contains entity URNs in an old and invalid format which causes exceptions. When I use the ElasticSearch APIs to clear this index, the error goes away, and when clicking around in the application it gets filled again, this time with data with valid URNs. Is it normal that this index isn't kept up to data with the normal entity deletes (as the database and non-hidden ElasticSearch indices are actually cleared)? If it is normal, is there a better way to clear the indices? Locally the clear of the hidden index works, but on our actual deployed environment this only clears some of the errors, but even with the deletes and the cleared index it still seems to find some invalid data somewhere?
📖 1
🔍 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
m
No, I didn't find a similar issue or a solution.
1. v0.10.0 (but I was also able to reproduce it in v0.10.2) 2. Quickstart & Kubernetes 3.
Copy code
2023-05-02 15:45:04,088 [pool-13-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Adatahub) - batchGet - 200 - 12ms
2023-05-02 15:45:04,094 [Thread-7219] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-elasticsearch.datahub-elasticsearch.svc.cluster.local:9200/corpuserindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-05-02 15:45:04,101 [Thread-7221] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-elasticsearch.datahub-elasticsearch.svc.cluster.local:9200/domainindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-05-02 15:45:04,114 [Thread-7222] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-elasticsearch.datahub-elasticsearch.svc.cluster.local:9200/datasetindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.datahub.graphql.GmsGraphQLEngine:1686 - Failed to load Entities of type: DataProduct, keys: [urn:li:dataProduct:(be.dcjm.infrastructuur-broninformatie-cji,PROD), urn:li:dataProduct:(be.kunsten.professionelekunsten-kunstenpunt-concerten,PROD), urn:li:dataProduct:(be.kunsten.professionelekunsten-theatervoorstellingen,PROD), urn:li:dataProduct:(be.publiq.vrijetijdsparticipatie-alle-activiteiten,PROD), urn:li:dataProduct:(be.dcjm.infrastructuur-erfgoedkaart-infrastructuur,PROD), urn:li:dataProduct:(be.acagroup.cji-cultuur-en-jeugdinfrastructuur,PROD), urn:li:dataProduct:(be.publiq.vrijetijdsparticipatie-publiq-uit-locaties,PROD), urn:li:dataProduct:(be.kunsten.locaties-kunstenpunt-locaties,PROD), urn:li:dataProduct:(be.dcjm.infrastructuur-kiosk-infrastructuur,PROD), urn:li:dataProduct:(be.vlaanderen.ovam.events-ovam-events,PROD), urn:li:dataProduct:(be.acagroup.cji-kiosk-infra,PROD), urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties] Failed to batch load data products
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
    ... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
    at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
    at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
    at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
    at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
    at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
    at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
    ... 3 common frames omitted
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
    ... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
    at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
    at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
    at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
    at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
    at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
    at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
    ... 3 common frames omitted
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
    ... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
    at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
    at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
    at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
    at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
    at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
    at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
    ... 3 common frames omitted
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
    ... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
    at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
    at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
    at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
    at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
    at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
    at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
    ... 3 common frames omitted
2023-05-02 15:45:04,175 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
    at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
    ... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
    at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
    at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
    at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService....
a
Hi, this could be an issue with soft-deletes, @dazzling-yak-93039 might be able to help here
m
I did the calls with
/openapi/entities/v1/?urns=urn%3Ali%3AdataProduct%3Abe.publiq.vrijetijdsparticipatie-alle-activiteiten&soft=false
so it should have been a hard delete.
a
Hi, is this issue still affecting you?
m
We recreated our complete Datahub from scratch (so data volumes were deleted) after which all data got pushed to Datahub again. So everything works now, but the issue itself isn't really resolved. If for another story/requirement we would need to change the makeup of a key we would run into this issue again and would again have to resort to recreating everything from scratch instead of being able to just delete some things.