Hi - we keep getting failures on two jobs we run daily via the datahub-upgrade component:

NoCodeDataMigrationCleanup

and

RestoreIndices

The failure for the first shows this in the logs:

Starting upgrade with id NoCodeDataMigrationCleanup...
Executing Step 1/4: UpgradeQualificationStep...
Found qualified upgrade candidate. Proceeding with upgrade...
Completed Step 1/4: UpgradeQualificationStep successfully.
Executing Step 2/4: DeleteLegacyAspectRowsStep...
Completed Step 2/4: DeleteLegacyAspectRowsStep successfully.
Executing Step 3/4: DeleteLegacyGraphRelationshipStep...
Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @b97c004)
Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @b97c004)
Failed Step 3/4: DeleteLegacyGraphRelationshipStep. Failed after 1 retries.
Exiting upgrade NoCodeDataMigrationCleanup with failure.
Upgrade NoCodeDataMigrationCleanup completed with result FAILED. Exiting...

For the second, it seems like it runs successfully for the most part, but has trouble emitting MCL for messages larger than the value of

max.request.size

, so I guess we can try to bump that up, but if you have any suggestions on this, let us know

Hi Hassan are you using Neo4j as your graph service?

Hey, no, we use elasticsearch

Hmm that’s a weird message to be getting

sorry for not getting back to you. We’re deploying using nomad to orchestrate the containers, which run on aws, and we’re using msk (aws kafka), opensearch (aws elasticsearch), postgres rds and glue for the schema registry