https://datahubproject.io logo
Join Slack
Powered by
Hello DataHub-Team! Since our latest ingestion run...
# troubleshoot
r
Hello DataHub-Team! Since our latest ingestion run, a duplicate key error occurs in GMS when trying to browse our Snowflake datasets. This renders DataHub unusable, because Snowflake is our primary data store. This is somehow related to Snowflake + dbt, but I could not narrow it down besides stating the obvious, that the Java code is trying to merge an equal object. Any help would be appreciated, because this is blocks our prod env. The following error occurs: In the Frontend UI: 
An unknown error occurred. (code 500)
Failed to load results! An unexpected error occurred.
And in the GMS logs, it looks like this:
Copy code
06:42:03.642 [Thread-484] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to retrieve entities of type Dataset
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
    at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1423)
    at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1144)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
    at org.dataloader.DataLoaderHelper.lambda$dispatchQueueBatch$3(DataLoaderHelper.java:272)
    at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
    at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Failed to retrieve entities of type Dataset
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$183(GmsGraphQLEngine.java:1588)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    ... 1 common frames omitted
Caused by: java.lang.RuntimeException: Failed to batch load Datasets
    at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:146)
    at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$183(GmsGraphQLEngine.java:1585)
    ... 2 common frames omitted
Caused by: java.lang.IllegalStateException: Duplicate key EntityAspectIdentifier(urn=urn:li:dataset:(urn:li:dataPlatform:dbt,XXXXXXXXXXXXX.YYYYYYYYYYYYYY.ZZZZZZZZZZZZZ,PROD), aspect=upstreamLineage, version=0) (attempted merging values com.linkedin.metadata.entity.EntityAspect@f613625b and com.linkedin.metadata.entity.EntityAspect@f613625b)
    at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:133)
    at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
    at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
    at java.base/java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:1033)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
    at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:263)
    at com.linkedin.metadata.entity.EntityService.getEnvelopedAspects(EntityService.java:1826)
    at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:379)
    at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:333)
    at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:289)
    at com.linkedin.metadata.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:109)
    at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:130)
    ... 3 common frames omitted
06:42:03.645 [Thread-421] ERROR c.datahub.graphql.GraphQLController:99 - Errors while executing graphQL query: "query getSearchResultsForMultiple($input: SearchAcrossEntitiesInput!) {\n  searchAcrossEntitie
...
restoring indices using the datahub-restore-indices-job did not help (also using the 
-a clean
option). Next thing I did was updating from v0.9.6.1 to v.0.10.0 but the error persists :-(
Some datasets are still working, others are not. i have not found a pattern yet.
Since we did not find any solution, we had to remove all the datasets from our catalog. Currently reingesting and hoping that this does not occur anymore. Any idea what may have caused such an error?
it occurs again, starting with our 2nd update of dbt. So, first dbt run is getting ingested successfully, but starting with the second it fails due to this "Duplicate key" error. What can be the cause of this? It's the very same dbt run, started from the same Airflow DAG.
The same error now prevents rendering half of my landing page, because the recommendation query fails. So I tried to narrow it down and my quest for finding the root cause ended here: This still works:
Copy code
query listRecommendations {
  listRecommendations(input: {userUrn:"urn:li:corpuser:ME", requestContext: {scenario: HOME}, limit: 4}) {
    modules {
      moduleId
      renderType
      content {
        value
        entity {
          ...on Entity {
            urn
          }
        }
      }
    }
  }
}
increasing the limit to 5 does not work anymore. Throws the same error as above, resulting in errors in the GraphQL query - the entities of type dataset are null:
Copy code
{
          "moduleId": "RecentlyEditedEntities",
          "renderType": "ENTITY_NAME_LIST",
          "content": [
            {
              "value": "urn:li:container:721c0d86a2715e294b582d01bcfeef89",
              "entity": {
                "urn": "urn:li:container:721c0d86a2715e294b582d01bcfeef89"
              }
            },
            {
              "value": "urn:li:dataset:(urn:li:dataPlatform:snowflake,abc,PROD)",
              "entity": null
            },
            {
              "value": "urn:li:dataset:(urn:li:dataPlatform:dbt,def,PROD)",
              "entity": null
            },
            {
              "value": "urn:li:container:d0748a3ef091458dda00a7df13b396df",
              "entity": {
                "urn": "urn:li:container:d0748a3ef091458dda00a7df13b396df"
              }
            }
          ]
        },
what's really strange, and here I do not comprehend it anymore, if I remove the fragment part of the query and use the urn directly, it works again, like this:
Copy code
query listRecommendations {
  listRecommendations(input: {userUrn:"urn:li:corpuser:ME", requestContext: {scenario: HOME}, limit: 5}) {
    modules {
      moduleId
      renderType
      content {
        value
        entity {
#          ...on Entity {
            urn
#          }
        }
      }
    }
  }
}
b
Hey @rich-pager-68736, This seems strange. Could you connect to the database behind datahub and query for the problematic urn?
Looking for ingestion time and differencies there.
r
Sure, can do that. Here's the export. Anything in particular I should query?
datahub_urn_records.csv
b
Try looking for 
urn=urn:li:dataset:(urn:li:dataPlatform:dbt,XXXXXXXXXXXXX.YYYYYYYYYYYYYY.ZZZZZZZZZZZZZ,PROD),
what caused the problem for you
r
The export from my previous reply contains one of the culprit datasets. So, this should be qualified as a bug example. And yes, I used a different pseudonymization, but all the AAAAAs refer to the very same table. dbt and snowflake are part of the export, because I queried by table name (using LIKE).
There is no obvious diverge from the norm from what I can tell. The table constraint are not violated if you thought in that direction, so no duplicates there.
l
Hi @rich-pager-68736 @microscopic-australia-66302, hope you’re well. I’m getting a similar problem to the one described above running version 0.10.1 of DataHub after running 
csv-enricher
ingestion that was meant to import some additional aspects - tags, glossary, an so on. I am wondering if by any chance you were able to find a satisfying solution to your problems and would share the solution with me (us :))
r
Hi @lively-spring-5482, I could not really identify the bug, but I could resolve the issue by purging our OpenSearch+MSK Cluster. Our database's integrity has been good all the time, so afterwards we could restore the indices and since then, the error has not occured anymore.
l
Thanks for the info! So it seems the problem is not in the relational database but somewhere in the indexes 🤔
Hi @rich-pager-68736 @microscopic-australia-66302 @big-carpet-38439 We have also experienced the same problem and made a small investigation. We were using csv-enricher ingesting data to Datahub v 0.10.1. Here are the findings: 1. We made the mistake to use flawed capitalisation in the platform name part of the URN (we input “*S*nowflake” not “snowflake”). 2. The script happily ingested flawed data creating new records in ElasticSearch 
datasetindex_v2
for something that got interpreted as new URNs. 3. When using the UI we’ve noticed lots of 500 errors when entering search prompt. 4. After an investigation we’ve noticed 
autoCompleteForMultiple
Graph API query is returning not only valid (existing) URNs but also those with incorrect capitalisation - see attachment. Note the list is not formally duplicated. 5. The GMS engine processing of 
autoCompleteForMultiple
involves retrieval of 
entities
aspect which is handled as a 
batchLoad
call from MySQL with the URNs mentioned in point 4 as input keys. 6. With: a. MySQL being case insensitive b. the UNION ALL logic used by batchLoad c. two ids being identical and differing only by their casing the result set retrieved from DAO contains duplicated entries. This can’t be processed further and GMS throws an exception when converting results to a map keyed by URN. 7. Ultimately we were able to solve the issue without clearing the whole database just by running a delete by query in ElasticSearch removing flawed “Snowflake” URNs from 
datasetindex_v2
. Not sure where the fix should be applied but there is clearly no validation of the input in csv-enricher and that eventually leads to serious data corruption. Also, there is a huge difference in the way the two persistence layers (ElasticSearch and MySQL) are behaving. This will no doubts lead to more problems in the future if not addressed.
plus1 2
20 Views
Open in Slack
PreviousNext
Powered by