Hi again, we’re still struggling with the datahub cron jobs failing on this method: https://github.com/datahub-project/datahub/blob/55357783f330950408e4624b3f1421594c[…]b/upgrade/nocodecleanup/DeleteLegacyGraphRelationshipsStep.java The failure being:

Starting upgrade with id NoCodeDataMigrationCleanup...
Executing Step 1/4: UpgradeQualificationStep...
Found qualified upgrade candidate. Proceeding with upgrade...
Completed Step 1/4: UpgradeQualificationStep successfully.
Executing Step 2/4: DeleteLegacyAspectRowsStep...
Completed Step 2/4: DeleteLegacyAspectRowsStep successfully.
Executing Step 3/4: DeleteLegacyGraphRelationshipStep...
Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7ca48474)
Failed to delete legacy data from graph: java.lang.ClassCastException: class com.linkedin.metadata.graph.elastic.ElasticSearchGraphService cannot be cast to class com.linkedin.metadata.graph.neo4j.Neo4jGraphService (com.linkedin.metadata.graph.elastic.ElasticSearchGraphService and com.linkedin.metadata.graph.neo4j.Neo4jGraphService are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @7ca48474)
Failed Step 3/4: DeleteLegacyGraphRelationshipStep. Failed after 1 retries.
Exiting upgrade NoCodeDataMigrationCleanup with failure.
Upgrade NoCodeDataMigrationCleanup completed with result FAILED. Exiting...

I’m not fully understanding this bit as it seems specific to neo4j, and I don’t know why that would/should be running for setups that aren’t using that. Thanks for you help in advance.

Hey @damp-greece-27806,

datahub cron jobs failing

Do you mean the datahub-ingestion-cron, that is defined in helm? Did you set `global.graph_service_impl`to elasticsearch?

hey - we don’t use kubernetes. We use Nomad to orchestrate docker containers, so we have basically translated the various docker-compose files to Nomad definitions. Nonetheless, the job I’m referring to comes from the datahub-upgrade image. We have the

GRAPH_SERVICE_IMPL

env var set to elasticsearch.

The cron job uses the datahub-ingstion image: https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L74

this is the docker image that is failing us: https://github.com/datahub-project/datahub/tree/master/docker/datahub-upgrade Are we not supposed to run this on a periodic/daily basis?

What's your goal with running it?

sorry for the delayed reponse. our datahub cluster was set up with datahub-upgrade running as a daily job. In truth, it didn’t make much sense as to why a job with

upgrade

in the name was something running in a scheduled basis. So is this job only run when we are upgrading versions? Either way it sounds like I can turn this into a manual job rather than a scheduled job Regarding datahub-ingestion, is that something we should run even though we can’t run datahub-actions (we use glue for our schema registry, I believe this isn’t supported due to AWS not having a sdk for certain glue APIs)?

upgrade jobs for upgrade yes, helm upgrade should run it. glue is a supported source: https://datahubproject.io/docs/generated/ingestion/sources/glue/

@damp-greece-27806 I think I know of your issue here. Have you recently upgraded from an old version of DataHub? It appears that this may be causing the Upgrade Step to fail here (and report this failure)

I’m pretty sure our migration path was

0.8.42

->

0.9.0

->

0.9.5

->

0.9.6

Hi @big-carpet-38439, I am facing the same issue as the OP’s first error message when tried to run clean up job. We upgraded from 0.8.45 to 0.10.0 (yes a big upgrade). We are using elastic search and not noe4j. We are facing following errors after the upgrade. 1. Error in UI

Validation error

(FieldUndefined@[listRecommen dations/modules/content/params/ searchParams/filters/value]):

Field 'value' in type 'FacetFilter' is undefined (code undefined)

So we tried below to troubleshoot 2) datahub-restore-indices job fails with

Java heap size OOM

Resource config for upgrade job

resources:

limits:

сри: 500m

memory: "'Gi"

requests:

сри: 300m

memory: "1Gi"

3) datahub-cleanup-job fails with error mentioned above Goal is to get rid of #1 error. We are fine to do a clean up

Could you paste the full stack traces for the restore indicies jobs and the upgrade job? Also, have you tried giving those jobs more memory, since often they may need a few GBs if you have a decent bit of metadata

@gray-shoe-75895, sorry for the delay in response. Here is a snippet of the error message for restore indicies job. I increased memory to 2Gi

Good to know that batch size helped! On the second one, it seems like it’s got some mixed configuration between Elasticsearch and neo4j as the graph backend. To confirm, you’re not using neo4j right?

Yes we are not using neo4j

I’m handling with the same issues, is there any solution for that? @gray-shoe-75895 @little-breakfast-38102

Are you still facing issues with v0.10.1?

For some reason, I am doing deploy to v0.10.1 and in the UI I see 0.10.0