Hello I was faced with error after updating datahub on k8s f DataHub #troubleshoot

Hello! I was faced with error after updating datah...

icy-flag-80360

03/29/2023, 12:10 PM

Hello! I was faced with error after updating datahub on k8s from version 0.10.0 to 0.10.1 with this exception:

Copy code

uppressed: org.elasticsearch.client.ResponseException: method [POST], host [<http://elasticsearch-master:9200>], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"[simple_query_string] analyzer [query_word_delimited] not found","index_uuid":"GZJPC-CBTtekUqWRmtZGfA","index":"datahubpolicyindex_v2"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"datahubpolicyindex_v2","node":"MhPwkRJ4T8WYHY0QwONrOg","reason":{"type":"query_shard_exception","reason":"[simple_query_string] analyzer [query_word_delimited] not found","index_uuid":"GZJPC-CBTtekUqWRmtZGfA","index":"datahubpolicyindex_v2"}}]},"status":400}

But if I'm check from curl in GMS pod - all ok, elastic returns data with existing policies, but without any index_uuid. Example: curl -XGET 'http://elasticsearch-master:9200/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true' Is any ways to repair it? I'll tried many ways to recover with full erasing elastic data too.

plus1 3

🚨 1

✅ 1

icy-flag-80360

03/30/2023, 9:48 AM

Workaround for this broken upgrade: 1. Stop datahub 0.10.1 2. Drop elasticsearch indices (for me this is:

curl -X DELETE "<http://localhost:9200/_all>"

) 3. Run MAE Consumer with version 0.9.6.1 (Last version of MAE, that create indices, if not created) 4. Run DatahubUpgrade 0.10.0 with SystemUpdate parameter ( "`java -jar /datahub/datahub-upgrade/bin/datahub-upgrade.jar -u SystemUpdate`") 5. After that run datahub with version 0.10 and run DatahubUpgrade with

java -jar /datahub/datahub-upgrade/bin/datahub-upgrade.jar -u RestoreIndices

In this moment as for me - 0.10.1 on k8s is broken=\

astonishing-answer-96712

04/03/2023, 8:39 PM

Thank you for the update here! I’ll see if the team can look into it!

orange-night-91387

04/03/2023, 9:22 PM

This sounds like you're running with custom helm charts? SystemUpdate runs automatically with the deployment and takes care of updating the indices. If you haven't updated your helm charts to the latest version, please do and make sure the upgrade job runs when you upgrade your deployment version.

icy-flag-80360

04/06/2023, 7:16 AM

@orange-night-91387 Yes, as I running chart on private k8s cluster - I was cutted most parts, and tried run Datahub Upgrade as init container, but it every time not help. And as I see new posts - everyone have same errors in 0.10.1 version - no new indexes in upgrade, and MAE just not create it. ElasticSearch version 7.13, that should be supported, if it matters.

98 Views

Open in Slack

Previous Next