Hi *, We've got an issue with broken searching and I'm out of ideas. 😞 This post is a follow-up to https://datahubspace.slack.com/archives/C02R2NBJXD1/p1674234240711219. The symptoms are that searching for strings contained in glossary term names results in correct displaying of specific individual glossary terms in the pull down list below the search field, but selecting "View all results for ..." (the searched string) or pressing Enter in the search field gives 'No results found for "..."' page and a HTTP error 400 in the GMS container log:

at com.linkedin.metadata.search.client.CachingEntitySearchService.search(CachingEntitySearchService.java:54)
	at com.linkedin.metadata.search.aggregator.AllEntitiesSearchAggregator.lambda$getSearchResultsForEachEntity$2(AllEntitiesSearchAggregator.java:161)
	at com.linkedin.metadata.utils.ConcurrencyUtils.lambda$transformAndCollectAsync$0(ConcurrencyUtils.java:24)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<http://elasticsearch:9200>], URI [/glossarytermindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
error={"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: Can't parse boolean value [architektura], expected [true] or [false]","index_uuid":"CDpqvb1FRB2F8rVTA53BVA","index":"glossarytermindex_v2"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"glossarytermindex_v2","node":"hYLiXc5_SCa0D2KGNC172Q","reason":{"type":"query_shard_exception","reason":"failed to create query: Can't parse boolean value [architektura], expected [true] or [false]","index_uuid":"CDpqvb1FRB2F8rVTA53BVA","index":"glossarytermindex_v2","caused_by":{"type":"illegal_argument_exception","reason":"Can't parse boolean value [architektura], expected [true] or [false]"}}}]} status=400

(there are more lines in the stack trace, but I believe that the most important lines are included this way). The string "architektura" appearing in the exception record at the bottom was my searched term. The important point is that we extended the Glossary Term entity by adding additional aspects with new attributes (as discussed elsewhere) including getting those new attributes supported in searching and filtering (I can share the PDL files for the added aspects here if necessary, of course). This happens on a completely new DataHub deployment of a forked repository (our last resync/merge includes commit a164bdab) with our modifications, i.e. both MySQL and Elastic Search databases were created from scratch (we used the option without Neo4J) and Glossary Terms were created anew after this deployment, i.e. there were no changes in the metamodel since the Elastic Search index was created. I tried restoring indices using the POST method provided within the GMS API for all URNs found in the MySQL database (as discussed elsewhere, I couldn't use the datahub-upgrade image due to our metamodel extensions), but that didn't help. Any help would be very appreciated...

Hi @gentle-portugal-21014, this could potentially be related to the structured search changes in 0.10.0. Did you recently upgrade?

Hi @astonishing-answer-96712, we did not upgrade in the sense of first having one version deployed and then changing the DataHub version while retaining the database and/or the metadata (we performed a new deployment with new database creation and all metadata created from the respective sources using ingestion). However, the commit number from your master branch mentioned above to which we synchronized our fork is in between 0.9.6 and 0.10.0 and thus some of the changes included in 0.10.0 are indeed included in our fork now. Do you have a particular commit in mind, so that we could check whether it's included in our fork/version?

@gentle-portugal-21014 This is very odd... Did you add any Searchable fields with boolean type?

@big-carpet-38439 There are three boolean fields in two different aspects, but none of them has enableAutocomplete set to true (I added that only to string attributes). However, I do have addToFilters set to true for those boolean attributes and also queryByDefault set to true for two of them. Do you think that removing queryByDefault might help (admittedly, I probably left it in the definition without thinking too much about it 😕 )? Moreover, I have queryByDefault set also for an enum attribute ("fieldType": "KEYWORD" in the Searchable annotation), but I assume that this shouldn't be causing issues there, right?

I think I would need the mappings on

glossarytermindex_v2/_mapping

index. Can we look at that for which field(s) are booleans. It almost seems like the query filter might be taking the query and searching for the value in these boolean fields. We would indeed look for cases where the mapping is boolean and the pdl model is

queryByDefault=true

Alright. Based on what we discussed today, I'll try to get a version without queryByDefault deployed and reindex. I had to create a script for reindexing by iterating through all unique urns found in the MySQL database with the GMS REST API call, because the datahub_upgrade docker image approach doesn't work with local metamodel extensions apparently, so I just hope that this way of reindexing works correctly. If that doesn't help, we can try changing the boolean in search annotations to keyword, deploy that (and reindex again), correct? I assume that I should change the search type of all the 3 boolean fields at once, right (i.e. it probably makes no sense to change just one of them)?