Hi, we have datahub running on an EKS cluster which is working well. We are currently experiencing an issue where whenever we restart the cluster or perform an upgrade all of the datahub ingestion secrets are wiped and we have to manually re-insert them into the UI. Does anyone know what the issue is here?

We've not seen this issue but this seems like a big one. Do you have access to your MySQL instance?

Yes there is an error in the logs as follows.

~~~~ Execution Summary ~~~~

RUN_INGEST - {'errors': [],
 'exec_id': 'ecfedd5c-728d-40e8-b009-904b79c438a8',
 'infos': ['2022-05-31 13:00:00.160258 [exec_id=ecfedd5c-728d-40e8-b009-904b79c438a8] INFO: Starting execution for task with name=RUN_INGEST',
           '2022-05-31 13:00:00.187733 [exec_id=ecfedd5c-728d-40e8-b009-904b79c438a8] INFO: Caught exception EXECUTING '
           'task_id=ecfedd5c-728d-40e8-b009-904b79c438a8, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 119, in execute_task\n'
           '    self.event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 81, in run_until_complete\n'
           '    return f.result()\n'
           '  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
           '    raise self._exception\n'
           '  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
           '    result = coro.send(None)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 74, in execute\n'
           '    recipe: dict = self._resolve_recipe(validated_args.recipe, ctx)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 147, in _resolve_recipe\n'
           '    raise TaskError(f"Failed to resolve secret with name {match}. Aborting recipe execution.")\n'
           'acryl.executor.execution.task.TaskError: Failed to resolve secret with name SNOWFLAKE_DEV_PASSWORD. Aborting recipe execution.\n']}
Execution finished with errors.

Are you deploying using helm?

we had a similar issue with airflow running in the same cluster which we solved by defining a fernet key, is there something similar in datahub?

If yes, are you using an older version of the Helm charts?

yes using helm

We had an issue previously (a few months ago) where each time an upgrade was performed, the encryption key for encrypting your secrets was regenerating on each deployment. This has been fixed however some versions back -- what is your Chart version?

the chart version we are using is 0.2.72

Okay and which version did you recently update from? From what it looks like, that one should have the fix, which originally went in early March -https://github.com/acryldata/datahub-helm/pull/93

this is the version that we originally deployed from

Oh - so you have not yet upgraded?

yeah we usually restart the cluster on upgrades

No -- this is "expected". We depend on state (secrets) that are persistent across redeployments of the pods

so we need to re-enter the secrets into the UI everytime we restart the cluster?

To provide your own encryption key secret (which is required for ingestion), you can disable the "provision_secret" flag here: https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L155 And simply provide our own by overriding this value: https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L151

i see i see that makes sense

It's so that we do not store your ingestion secrets in plaintext. But we also do not provide a "default" encryption key because of inherent risk that comes with

okay that makes sense, we will give this a try. I'll let you know how it goes, thanks so much

ok sounds good! definitely let me know!