Hello everyone! We are trying to upgrade from 0.9....
# troubleshoot
w
Hello everyone! We are trying to upgrade from 0.9.6 to 0.10.1 but faced with 500 error in UI Logs from GMS (similar with datahub-datahub-system-update-job logs):
Copy code
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://123.123.123.123:9200], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":942}],"type":"x_content_parse_exception","reason":"[1:942] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:942] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:942] [bool] failed to parse field [should]","caused_by":{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":942}}}},"status":400}
How we can fix that? Thanks a lot! PS ElasticSearch version: 7.9.3 HelmChart version: datahub-0.2.161
ScreenShot 2023-04-05 в 15.20.45.png
Also
datahub-elasticsearch-setup-job
logs here:
Copy code
2023/04/05 09:29:32 Waiting for: http://123.123.123.123:9200
2023/04/05 09:29:32 Received 200 from http://123.123.123.123:9200
going to use protocol: http
going to use default elastic headers
not using any prefix

 datahub_analytics_enabled: true

>>> GET _opendistro/_ism/policies/datahub_usage_event_policy response code is 404
>>> creating _opendistro/_ism/policies/datahub_usage_event_policy because it doesn't exist ...
{
  "policy": {
    "policy_id": "datahub_usage_event_policy",
    "description": "Datahub Usage Event Policy",
    "default_state": "Rollover",
    "schema_version": 1,
    "states": [
      {
        "name": "Rollover",
        "actions": [
          {
            "rollover": {
              "min_index_age": "1d"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "ReadOnly",
            "conditions": {
              "min_index_age": "7d"
            }
          }
        ]
      },
      {
        "name": "ReadOnly",
        "actions": [
          {
            "read_only": {}
          }
        ],
        "transitions": [
          {
            "state_name": "Delete",
            "conditions": {
              "min_index_age": "60d"
            }
          }
        ]
      },
      {
        "name": "Delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": [
        "datahub_usage_event-*"
      ],
      "priority": 100
    }
  }
}{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Invalid field: [ism_template] found in Policy."}],"type":"illegal_argument_exception","reason":"Invalid field: [ism_template] found in Policy."},"status":400}
>>> GET _template/datahub_usage_event_index_template response code is 200
>>> _template/datahub_usage_event_index_template already exists ✓

>>> GET datahub_usage_event-000001 response code is 200
>>> datahub_usage_event-000001 already exists ✓
2023/04/05 09:29:32 Command finished successfully.
b
w
yeap, thanks for you information i think about it, but i didn't find information about this in datahub documentation I will try to deploy ES with 7.10.2 version and come back with the result
hi i updated version of ES to 7.10.2 but faced with new issue UI:
Copy code
Failed to perform post authentication steps. Error message: Failed to provision user with urn urn:li:corpuser:shershov-d.
Also part of GMS logs here:
Copy code
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
and datahub-actions pod with:
Copy code
2023/04/06 13:42:28 Problem with request: Get "http://datahub-datahub-gms:8080/health": dial tcp 10.123.123.123:8080: connect: connection refused. Sleeping 1s
2023/04/06 13:42:28 Timeout after 4m0s waiting on dependencies to become available: [http://datahub-datahub-gms:8080/health]
b
That's a different error now, but seems like gms is giving some error. What does the log say
^abit weird for gms to be complaining about 8080 not accessible
a
Hi, @dazzling-yak-93039 may be able to help you with this next week!
w
Hi! So, finally I tried to migrate elastic from version 7.9 to version 7.10 by migrate all indexes with elasticdump/elasticsearch-dump tool, because that datahub cannot be run on versions below 7.10 after migration I see that data is absent by No Metadata Found message. I need any ideas about this situation and I am hope for your help
b
Run the reindexing job in helm?
w
yeap, you mean elasticsearchSetupJob?
b
If you deploy the helm chart there should be a Job template that you can run
Elasticsearch setup doesn't reindex - it just sets up some settings in a new es cluster
Lemme find the doc for reindex
w
hmm, How i can run this job?
okay, okay thanks a lot! i will try!
b
Yes that's the one
w
great! i will try! Thanks a lot!
So, I was able to run ES job on restore-indices, that is really working, but I think that I don't have fully data after migration Datasets are empty Domain and glossary list was empty too
b
@astonishing-answer-96712 @dazzling-yak-93039 maybe you can advise better on this
w
Also, for information, when I started cleanup Job I got an error:
Copy code
Completed Step 1/4: UpgradeQualificationStep successfully.
Executing Step 2/4: DeleteLegacyAspectRowsStep...
Completed Step 2/4: DeleteLegacyAspectRowsStep successfully.
Executing Step 3/4: DeleteLegacyGraphRelationshipStep...
Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @61a485d2)
Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @61a485d2)
Failed Step 3/4: DeleteLegacyGraphRelationshipStep. Failed after 1 retries.
Exiting upgrade NoCodeDataMigrationCleanup with failure.
Upgrade NoCodeDataMigrationCleanup completed with result FAILED. Exiting...
But datahub-restore-indices-adhoc Job was working fine
oh, it's working! I restarted cleanup job - another failed, but after I started another job datahub-restore-indices-adhoc with success status and I see that all data was successfully migrated! @better-orange-49102 thank you for idea with restore-indices!