many-glass-1784
05/04/2023, 3:03 PMCaused by: java.lang.RuntimeException: Failed to batch load data products
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
... 3 common frames omitted
After some looking around I found that there was some data in a hidden ElasticSearch index, .ds-datahub_usage_event-000001, that still seemed to contain rows with URNs that refer to data that was deleted. it looks like the recommendations/suggestions section on the main page uses this index and as such gets some data that contains entity URNs in an old and invalid format which causes exceptions.
When I use the ElasticSearch APIs to clear this index, the error goes away, and when clicking around in the application it gets filled again, this time with data with valid URNs.
Is it normal that this index isn't kept up to data with the normal entity deletes (as the database and non-hidden ElasticSearch indices are actually cleared)? If it is normal, is there a better way to clear the indices? Locally the clear of the hidden index works, but on our actual deployed environment this only clears some of the errors, but even with the deletes and the cleared index it still seems to find some invalid data somewhere?lively-cat-88289
05/04/2023, 3:04 PMmany-glass-1784
05/04/2023, 3:05 PMmany-glass-1784
05/04/2023, 3:09 PM2023-05-02 15:45:04,088 [pool-13-thread-1] INFO c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Adatahub) - batchGet - 200 - 12ms
2023-05-02 15:45:04,094 [Thread-7219] WARN org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-elasticsearch.datahub-elasticsearch.svc.cluster.local:9200/corpuserindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-05-02 15:45:04,101 [Thread-7221] WARN org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-elasticsearch.datahub-elasticsearch.svc.cluster.local:9200/domainindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-05-02 15:45:04,114 [Thread-7222] WARN org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-elasticsearch.datahub-elasticsearch.svc.cluster.local:9200/datasetindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.datahub.graphql.GmsGraphQLEngine:1686 - Failed to load Entities of type: DataProduct, keys: [urn:li:dataProduct:(be.dcjm.infrastructuur-broninformatie-cji,PROD), urn:li:dataProduct:(be.kunsten.professionelekunsten-kunstenpunt-concerten,PROD), urn:li:dataProduct:(be.kunsten.professionelekunsten-theatervoorstellingen,PROD), urn:li:dataProduct:(be.publiq.vrijetijdsparticipatie-alle-activiteiten,PROD), urn:li:dataProduct:(be.dcjm.infrastructuur-erfgoedkaart-infrastructuur,PROD), urn:li:dataProduct:(be.acagroup.cji-cultuur-en-jeugdinfrastructuur,PROD), urn:li:dataProduct:(be.publiq.vrijetijdsparticipatie-publiq-uit-locaties,PROD), urn:li:dataProduct:(be.kunsten.locaties-kunstenpunt-locaties,PROD), urn:li:dataProduct:(be.dcjm.infrastructuur-kiosk-infrastructuur,PROD), urn:li:dataProduct:(be.vlaanderen.ovam.events-ovam-events,PROD), urn:li:dataProduct:(be.acagroup.cji-kiosk-infra,PROD), urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties] Failed to batch load data products
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
... 3 common frames omitted
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
... 3 common frames omitted
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
... 3 common frames omitted
2023-05-02 15:45:04,174 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:393)
at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:336)
at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:292)
at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:111)
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:63)
... 3 common frames omitted
2023-05-02 15:45:04,175 [Thread-7224] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type DataProduct
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1687)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to batch load data products
at com.linkedin.datahub.graphql.types.dataproduct.DataProductType.batchLoad(DataProductType.java:81)
at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$194(GmsGraphQLEngine.java:1684)
... 2 common frames omitted
Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields do not have same length for urn:li:dataProduct:be.publiq.vrijetijdsparticipatie-publiq-uit-locaties
at com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:95)
at com.linkedin.metadata.entity.EntityService.getKeyEnvelopedAspect(EntityService.java:1891)
at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService....astonishing-answer-96712
05/05/2023, 5:44 PMmany-glass-1784
05/08/2023, 8:01 AM/openapi/entities/v1/?urns=urn%3Ali%3AdataProduct%3Abe.publiq.vrijetijdsparticipatie-alle-activiteiten&soft=false so it should have been a hard delete.astonishing-answer-96712
05/16/2023, 9:44 PMmany-glass-1784
05/17/2023, 10:09 AM