Hi team. I'm running the quickstart image, and aft...
# troubleshoot
j
Hi team. I'm running the quickstart image, and after a restart, datahub-gms is went unhealthy, and it is in that state ever since. In the logs I found this: at java.base/java.lang.Thread.run(Thread.java:829) Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":943}],"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [should]","caused_by":{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":943}}}},"status":400} Is it possible to fix this somehow?
g
Datahub version ?
j
Where can I check that? docker image ls only shows this: linkedin/datahub-frontend-react head 395976519aba 33 hours ago 729MB linkedin/datahub-gms head 1ea792edff08 33 hours ago 406MB linkedin/datahub-kafka-setup head b509103a08c2 33 hours ago 717MB acryldata/datahub-mysql-setup head 2741b96e1683 33 hours ago 53.1MB linkedin/datahub-elasticsearch-setup head 9bbe154e9af1 33 hours ago 23.3MB
g
datahub —version
j
0.9.2.4
g
latest is v0.10.0
if it is your testing/dev instance, then you can perform below steps to clean data and restart again • datahub docker nuke • docker system prune • datahub docker quickstart
j
it is a dev instance, but I rather not loose data if possible
g
Could you please share debug and info log from container You can find documentation at https://datahubproject.io/docs/how/extract-container-logs/
j
I'll get the logs and get back to you in a few hours (I have a few meetings incoming), thank you for your help
Hello, sorry, my friday was a bit chaotic. I attached the log files.
gmslog.zip
Any update on this? I've found this in the debug log:
2023-03-27 061526,085 [pool-10-thread-1] DEBUG c.l.m.s.e.q.r.SearchRequestHandler - Search request is: SearchRequest{searchType=QUERY_THEN_FETCH, indices=[], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":30,"query":{"bool":{"must":[{"function_score":{"query":{"bool":{"should":[{"bool":{"should":[{"simple_query_string":{"query":"*","fields":["urn^10.0","displayName^1.0","description^1.0"],"analyzer":"keyword","flags":-1,"default_operator":"and","analyze_wildcard":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"fuzzy_transpositions":true,"boost":1.0}},{"simple_query_string":{"query":"*","fields":["displayName.delimited^0.4","description.delimited^0.4","urn.delimited^5.0"],"analyzer":"query_word_delimited","flags":-1,"default_operator":"and","analyze_wildcard":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"should":[{"term":{"urn":{"value":"*","boost":100.0,"_name":"urn"}}},{"term":{"urn":{"value":"*","case_insensitive":true,"boost":70.0,"_name":"urn"}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"functions":[{"filter":{"match_all":{"boost":1.0}},"weight":1.0}],"score_mode":"avg","boost_mode":"multiply","max_boost":3.4028235E38,"boost":1.0}},{"bool":{"must_not":[{"match":{"removed":{"query":true,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":["urn"],"excludes":[]},"sort":[{"lastUpdatedTimestamp":{"order":"desc"}},{"urn":{"order":"asc"}}]}} 2023-03-27 061526,085 [pool-10-thread-1] DEBUG c.l.m.s.e.query.ESSearchDAO - Executing request 1679897726085: SearchRequest{searchType=QUERY_THEN_FETCH, indices=[datahubpolicyindex_v2], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":30,"query":{"bool":{"must":[{"function_score":{"query":{"bool":{"should":[{"bool":{"should":[{"simple_query_string":{"query":"*","fields":["urn^10.0","displayName^1.0","description^1.0"],"analyzer":"keyword","flags":-1,"default_operator":"and","analyze_wildcard":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"fuzzy_transpositions":true,"boost":1.0}},{"simple_query_string":{"query":"*","fields":["displayName.delimited^0.4","description.delimited^0.4","urn.delimited^5.0"],"analyzer":"query_word_delimited","flags":-1,"default_operator":"and","analyze_wildcard":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"should":[{"term":{"urn":{"value":"*","boost":100.0,"_name":"urn"}}},{"term":{"urn":{"value":"*","case_insensitive":true,"boost":70.0,"_name":"urn"}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"functions":[{"filter":{"match_all":{"boost":1.0}},"weight":1.0}],"score_mode":"avg","boost_mode":"multiply","max_boost":3.4028235E38,"boost":1.0}},{"bool":{"must_not":[{"match":{"removed":{"query":true,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":["urn"],"excludes":[]},"sort":[{"lastUpdatedTimestamp":{"order":"desc"}},{"urn":{"order":"asc"}}]}} 2023-03-27 061526,091 [pool-10-thread-1] DEBUG c.l.m.s.e.query.ESSearchDAO - Returning from request 1679897726085.
I guess it has something to do with this enrtry in the gms log
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":943}],"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [should]","caused_by":{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":943}}}},"status":400}
w
Hi. I'm encountering what looks like this same problem as well. My entrypoint is when I am running the
SystemUpdate
command (we're upgrading from 0.8.35 to 0.10.1). Very interested in hearing any tips on how to solve this.
Seems in my case our elasticsearch (AWS Opensearch) is running version 7.9 which doesn't support "case_insensitive". 7.17 seems to support it though.
n
We’re facing the same issue as Thomas’s. To make things even more interesting, we have the exact upgrade scenario (SystemUpdate from 0.8.35 to 0.10.1) and ES version (7.9).
w
@numerous-byte-87938 I simply upgraded my elastic to 7.10.