https://datahubproject.io logo
Join Slack
Powered by
Hi! I am experiencing the following problem with D...
# all-things-deployment
s
Hi! I am experiencing the following problem with DataHub cleanup job(logs in 🧵) which is deployed in k8s through Helm charts on version 0.10.3
Copy code
2023-06-04 04:59:02,515 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Completed Step 1/4: UpgradeQualificationStep successfully.
2023-06-04 04:59:02,515 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Executing Step 2/4: DeleteLegacyAspectRowsStep...
2023-06-04 04:59:02,705 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Completed Step 2/4: DeleteLegacyAspectRowsStep successfully.
2023-06-04 04:59:02,706 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Executing Step 3/4: DeleteLegacyGraphRelationshipStep...
2023-06-04 04:59:02,708 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @61a485d2)
2023-06-04 04:59:02,708 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @61a485d2)
2023-06-04 04:59:02,709 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Failed Step 3/4: DeleteLegacyGraphRelationshipStep. Failed after 1 retries.
2023-06-04 04:59:02,709 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Exiting upgrade NoCodeDataMigrationCleanup with failure.
2023-06-04 04:59:02,710 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Upgrade NoCodeDataMigrationCleanup completed with result FAILED. Exiting...
Can someone help me on how to resolve this issue?
a
Hi, @brainy-tent-14503 can likely speak to this
b
I am not sure why it is trying to execute a neo4j function on elasticsearch. I’ve created a check to prevent this condition here.
s
This would be fixed from 0.10.4 Thanks
Hi again! I have just upgraded to v0.10.4.1.. now I start seeing a different error message
Copy code
2023-06-15 21:45:10,503 [main] ERROR c.l.d.u.impl.DefaultUpgradeReport:22 - Failed to delete legacy search index: %s
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=illegal_argument_exception, reason=Wildcard expressions or all indices are not allowed]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572)
	at org.elasticsearch.client.IndicesClient.delete(IndicesClient.java:114)
	at com.linkedin.datahub.upgrade.nocodecleanup.DeleteLegacySearchIndicesStep.lambda$executable$1(DeleteLegacySearchIndicesStep.java:43)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
	at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
	at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:80)
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
	at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:23)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
	Suppressed: org.elasticsearch.client.ResponseException: method [DELETE], host [<https://my-opensearch-service.com:11862>], URI [/*document*?master_timeout=30s&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true&ignore_throttled=false&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Wildcard expressions or all indices are not allowed"}],"type":"illegal_argument_exception","reason":"Wildcard expressions or all indices are not allowed"},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
		... 22 common frames omitted
2023-06-15 21:45:10,504 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Failed Step 4/4: DeleteLegacySearchIndicesStep. Failed after 1 retries.
2023-06-15 21:45:10,504 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Exiting upgrade NoCodeDataMigrationCleanup with failure.
2023-06-15 21:45:10,589 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Upgrade NoCodeDataMigrationCleanup completed with result FAILED. Exiting...
2023-06-15 21:45:10,595 [Thread-0] WARN  i.e.d.pool.PooledConnectionQueue:384 - Closing busy connections on shutdown size: 1
b
I’ll follow up internally on this. I am inclined to delete this step. Its old and not sure it even serves a purpose. It looks like this probably was failing silently until recently.
Open in Slack
PreviousNext
Powered by