Hi Team, We are facing an exception while performing customProperties search in Datahub UI, for few specific properties. It gives 500, unknown error in PROD env. Expected to get list of entities for the above search , like we receive for all other property search, where as getting exception for few properties. search command we tried :- customProperties:<filedname> / customProperties:<fieldname>=<value>* Issue details 1. customProperties search works in STG and DEV env for all properties 2. customProperties search throws exception in PROD env only for few specific properties, and works for rest of the properties First screenshot shows the exception received in PROD and the second screenshot shows the same command that worked in STG env. Could you please guide us to resolve this. cc: @melodic-match-38516

We’ve been seeing issues with complex search in the latest release- what version are you on?

@astonishing-answer-96712 We are on Datahub v0.9.2

Hi @astonishing-answer-96712 Any insight you could give would be very much appreciated. We can’t for the life of us find the cause. Our DEV, STG and PROD are all based on the same IaaC so there should be no difference between the environments.

Hi Y’all, thank you for the patience from the long weekends here- if you are using quickstart, could you try specifying your version with this flag:

datahub docker quickstart --version=v0.X.X

Thank You @astonishing-answer-96712, let us check on the above and confirm

Ok, how is your instance deployed then?

@astonishing-answer-96712 We have deployed datahub in a kubernetes cluster.

Hi @astonishing-answer-96712 , can we have some advice on this?

@brainy-tent-14503 may be able to help!

This looks like the logs from the frontend? Are you able to share the gms logs?

@brainy-tent-14503 this is the log for GMS

These are the gms logs when running the search that results in the error? The second one is definitely not search related, looks like a series of counts. I would expect an exception and stacktrace that corresponds with the failed search. It may be before or after these lines?

@brainy-tent-14503 Attached is the latest log where we generated the same exception. (exception occurs from 5:01 timestamp, followed by stack trace)

I did take a look at this log and I do not see a stack trace anywhere in the file unfortunately.

@brainy-tent-14503 Sorry , could you please check for "Failed to execute DataFetcher" in the log shared

@brainy-tent-14503 Here is one example from the log

05:01:24.016 [ForkJoinPool.commonPool-worker-41] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type Dataset
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1506)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.RuntimeException: Failed to batch load Datasets
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:145)
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1503)
	... 7 common frames omitted
Caused by: java.lang.IllegalArgumentException: No enum constant com.linkedin.datahub.graphql.generated.FabricType.$UNKNOWN
	at java.base/java.lang.Enum.valueOf(Enum.java:240)
	at com.linkedin.datahub.graphql.generated.FabricType.valueOf(FabricType.java:8)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.mapDatasetKey(DatasetMapper.java:111)
	at com.linkedin.datahub.graphql.types.common.mappers.util.MappingHelper.mapToResult(MappingHelper.java:22)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.apply(DatasetMapper.java:75)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.map(DatasetMapper.java:61)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.lambda$batchLoad$0(DatasetType.java:141)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:143)
	... 8 common frames omitted
05:01:24.019 [ForkJoinPool.commonPool-worker-41] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type Dataset
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1506)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.RuntimeException: Failed to batch load Datasets
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:145)
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1503)
	... 7 common frames omitted
Caused by: java.lang.IllegalArgumentException: No enum constant com.linkedin.datahub.graphql.generated.FabricType.$UNKNOWN
	at java.base/java.lang.Enum.valueOf(Enum.java:240)
	at com.linkedin.datahub.graphql.generated.FabricType.valueOf(FabricType.java:8)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.mapDatasetKey(DatasetMapper.java:111)
	at com.linkedin.datahub.graphql.types.common.mappers.util.MappingHelper.mapToResult(MappingHelper.java:22)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.apply(DatasetMapper.java:75)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.map(DatasetMapper.java:61)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.lambda$batchLoad$0(DatasetType.java:141)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:143)
	... 8 common frames omitted
05:01:24.019 [ForkJoinPool.commonPool-worker-41] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type Dataset
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1506)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.RuntimeException: Failed to batch load Datasets
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:145)
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1503)
	... 7 common frames omitted
Caused by: java.lang.IllegalArgumentException: No enum constant com.linkedin.datahub.graphql.generated.FabricType.$UNKNOWN
	at java.base/java.lang.Enum.valueOf(Enum.java:240)
	at com.linkedin.datahub.graphql.generated.FabricType.valueOf(FabricType.java:8)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.mapDatasetKey(DatasetMapper.java:111)
	at com.linkedin.datahub.graphql.types.common.mappers.util.MappingHelper.mapToResult(MappingHelper.java:22)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.apply(DatasetMapper.java:75)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.map(DatasetMapper.java:61)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.lambda$batchLoad$0(DatasetType.java:141)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:143)
	... 8 common frames omitted
05:01:24.020 [ForkJoinPool.commonPool-worker-41] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Failed to retrieve entities of type Dataset
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1506)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.RuntimeException: Failed to batch load Datasets
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:145)
	at com.linkedin.datahub.graphql.GmsGraphQLEngine.lambda$createDataLoader$175(GmsGraphQLEngine.java:1503)
	... 7 common frames omitted
Caused by: java.lang.IllegalArgumentException: No enum constant com.linkedin.datahub.graphql.generated.FabricType.$UNKNOWN
	at java.base/java.lang.Enum.valueOf(Enum.java:240)
	at com.linkedin.datahub.graphql.generated.FabricType.valueOf(FabricType.java:8)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.mapDatasetKey(DatasetMapper.java:111)
	at com.linkedin.datahub.graphql.types.common.mappers.util.MappingHelper.mapToResult(MappingHelper.java:22)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.apply(DatasetMapper.java:75)
	at com.linkedin.datahub.graphql.types.dataset.mappers.DatasetMapper.map(DatasetMapper.java:61)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.lambda$batchLoad$0(DatasetType.java:141)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at com.linkedin.datahub.graphql.types.dataset.DatasetType.batchLoad(DatasetType.java:143)
	... 8 common frames omitted

This seems like you might have some bad data. Specifically ones where the FabricType is not a valid enum, specified in this model definition. You probably need to find and delete the invalid dataset(s).

They're on v0.9.2 which might be before the validation fixes went in. I see 3 invalid URNs in the log:

(urn:li:dataPlatform:bigquery,dataset.bq.fr-prd-datalake.prd_dlk_aeg_loaded_001.daily_weather_actual_jwa,PRD)

urn:li:dataset:(urn:li:dataPlatform:bigquery,dataset.bq.fr-prd-datalake.prd_dlk_aeg_loaded_001.daily_weather_forecast_jwa,PRD)

urn:li:dataset:(urn:li:dataPlatform:bigquery,dataset.bq.fr-prd-datalake.<http://prd_dlk_aeg_loaded_001.fr|prd_dlk_aeg_loaded_001.fr>_r_loc_sku_dy_act,PRD)

Note the PRD FabricTypes. These should not be ingestable in the current release

@orange-night-91387 @brainy-tent-14503 Thanks a lot for the update. So any version upgrade is required to fix this search issue? Regarding the invalid URNS :- We tried deleting the entities. I guess we could remove them one by one using the URN. Although we tried deleting these entities in one GO using the command listed in the metadata documentation , based on --env PRD , it didn't worked. --env PRD, listed 0 entities. Can you suggest any other way to achieve this?

Upgrading won't fix the bad data, but should prevent future bad data. If you have access it is probably easier to remove the bad entries at the data source layer directly (i.e. MySQL & ElasticSearch)

@brainy-tent-14503 @orange-night-91387 Thank you very much. @busy-analyst-35820 Let’s talk offline once.

Chris - were you able to get this sorted out?