Hi all, After I enabled the metadata authentication in DataHub, my DataHub Ingestions couldn't access the Secrets created for Ingestions. This isn't the an issue in my localhost deployment of Datahub using the quickstart. Can you please suggestion how to fix this?

~~~~ Execution Summary ~~~~

RUN_INGEST - {'errors': [],
 'exec_id': 'efbfe9b7-fd83-4a37-bff7-d6fd3e2186dc',
 'infos': ['2022-09-28 11:44:07.154056 [exec_id=efbfe9b7-fd83-4a37-bff7-d6fd3e2186dc] INFO: Starting execution for task with name=RUN_INGEST',
           '2022-09-28 11:44:07.182384 [exec_id=efbfe9b7-fd83-4a37-bff7-d6fd3e2186dc] INFO: Caught exception EXECUTING '
           'task_id=efbfe9b7-fd83-4a37-bff7-d6fd3e2186dc, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n'
           '    self.event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 89, in run_until_complete\n'
           '    return f.result()\n'
           '  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
           '    raise self._exception\n'
           '  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
           '    result = coro.send(None)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 67, in execute\n'
           '    recipe: dict = SubProcessTaskUtil._resolve_recipe(validated_args.recipe, ctx, self.ctx)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_task_common.py", line 84, in _resolve_recipe\n'
           '    raise TaskError(f"Failed to resolve secret with name {match}. Aborting recipe execution.")\n'
           'acryl.executor.execution.task.TaskError: Failed to resolve secret with name Dev_DataHub_PRIVATE_KEY_ID. Aborting recipe execution.\n']}
Execution finished with errors.

Could you try deleting and reconfiguring the ingestion setup?

I've tried deleting and recreating the same ingestions but the ingestions created from UI also couldn't read the Ingestion secrets. Also tried re-creating the secrets but noting works. These ingestions were previously running successfully when I didn't enable the metadata service authentication on deployment.

i do not use ui ingestion.

@early-airplane-84388 could we set up a call sometime next week to dive into it?