Hello team. In the context of upgrade tests on datahub, I have to restore the mysql data (67Mb), which works without any problem. But as indicated here, https://datahubproject.io/docs/how/restore-indices/, I also need to restore the indices, with the datahub-restore-indices-adhoc job. On a node with 50GB of RAM, this job fails, with the following error:

java.lang.OutOfMemoryError: Java heap space

I am using datahub-helm 0.2.89 and I have to start from this version. Any idea ? Thanks.

Have you tried increasing the upper limit on the resources in your values.yaml configuration? Specifically here, the cpu and memory requirements are specified.

restoreIndices:
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 300m
        memory: 256Mi

I would try something like this in your values, specifically changing the 512Mi to 2Gi

datahubUpgrade:  
  restoreIndices:
    resources:
      limits:
        cpu: 500m
        memory: 2Gi
      requests:
        cpu: 300m
        memory: 256Mi

Same error. Note that restoreIndices only appears in the values file from version 0.2.91 of datahub helm chart (I need to restore from version 0.2.89).

The template for the job should exist and be editable only manually in older releases. Like

kubectl -n <namespace> get cronjobs

should return something like this

dh-datahub-restore-indices-job-template

depending on your release name. You can use kubectl to edit this template manually since the older helm chart doesn’t allow this setting.

Thank you very much, the script ran well.

Restore indices is creating a lot of cloned indices in Elastic Search and thats causing the increase in search latency for our ES. Is there way to clean up these indices ?

The cloned indices are created by the

datahub-upgrade

system update job during installation and upgrades. These clones are automatically removed after 7 days. At the end of a

datahub-update

job it will look for and remove these.

Thanks @brainy-tent-14503 they were not removed we have cloned indices as old as 60 days now by any chance is

datahub-datahub-system-update-job

is the one who takes care of it and yes then how can I configure it to clean 7 days old indices

@powerful-memory-77948 - Apologies, I believe we internally lowered it to 7 days. The environment variable is here,

ELASTICSEARCH_BUILD_INDICES_RETENTION_VALUE

&

ELASTICSEARCH_BUILD_INDICES_RETENTION_UNIT

and is defaulted to 60 days.