Hey team, I am trying to delete old ingestions from old installment of DataHub while executing the command from the datahub-acryl k8s pod. Remnants of the old installation are still in ElasticSearch/mysql, how can I delete these old ingestions? using

datahub delete --env PROD --entity_type dataset

gives me assertion error:

File "/usr/local/lib/python3.10/site-packages/datahub/cli/cli_utils.py", line 403, in get_urns_by_filter
    329  def get_urns_by_filter(
    330      platform: Optional[str],
    331      env: Optional[str] = None,
    332      entity_type: str = "dataset",
    333      search_query: str = "*",
    334      include_removed: bool = False,
    335      only_soft_deleted: Optional[bool] = None,
    336  ) -> Iterable[str]:
 (...)
    399          for x in results["value"]["entities"]:
    400              entities_yielded += 1
    401              log.debug(f"yielding {x['entity']}")
    402              yield x["entity"]
--> 403          assert (
    404              entities_yielded == num_entities
    ..................................................
     platform = 'mysql'
     Optional = typing.Optional
     env = None
     entity_type = 'dataset'
     search_query = '*'
     include_removed = False
     only_soft_deleted = None
     Iterable = typing.Iterable
     results = {'value': {'numEntities': 1978,
                          'pageSize': 10000,
                          'from': 0,
                          'metadata': {...},
                          'entities': [...]}}
     entities_yielded = 0
     log.debug = <method 'Logger.debug' of <Logger datahub.cli.cli_utils (DEBUG)> __init__.py:1455>
     num_entities = 1978
    ..................................................

---- (full traceback above) ----
File "/usr/local/lib/python3.10/site-packages/datahub/entrypoints.py", line 149, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/upgrade/upgrade.py", line 386, in async_wrapper
    loop.run_until_complete(run_func_check_upgrade())
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
File "/usr/local/lib/python3.10/site-packages/datahub/upgrade/upgrade.py", line 373, in run_func_check_upgrade
    ret = await the_one_future
File "/usr/local/lib/python3.10/site-packages/datahub/upgrade/upgrade.py", line 366, in run_inner_func
    return await loop.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 343, in wrapper
    raise e
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 295, in wrapper
    res = func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/cli/delete_cli.py", line 192, in delete
    deletion_result = delete_with_filters(
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 343, in wrapper
    raise e
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 295, in wrapper
    res = func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/cli/delete_cli.py", line 245, in delete_with_filters
    urns = list(
File "/usr/local/lib/python3.10/site-packages/datahub/cli/cli_utils.py", line 403, in get_urns_by_filter
    assert (

AssertionError: Did not delete all entities, try running this command again!

hey Natanhel! just wanted to let you know we're looking into this and will be back to you shortly

Hi @cool-kitchen-48091, it seems like there might be something more going on here, but in the meantime if you’d like to try, we also have ingestion run rollbacks, see here: https://datahubproject.io/docs/how/delete-metadata/#rollback-ingestion-batch-run

Hey - we’ve seen this issue on our end as well - we have a fix merged that will be included in the next release of the cli

thanks, ran the delete command and nothing happen so what should we run after upgrading to the merged version?

The merged version hasn’t been released yet, but will be later today. The same delete command should work once you upgrade acryl-datahub

Can we have a link to the PR so we can follow the progress? Thanks!

It’s been released as part of acryl-datahub 0.9.5.2 🙂

Hi, version 0.9.5.2 also gives the error;AssertionError: Did not delete all entities, try running this command again! [2023-01-12 073436,790] DEBUG {datahub.entrypoints:198} - DataHub CLI version: 0.9.0.5 at /home/datahub/.local/lib/python3.9/site-packages/datahub/__init__.py [2023-01-12 073436,790] DEBUG {datahub.entrypoints:201} - Python version: 3.9.9 (main, Dec 21 2021, 100334) [GCC 10.2.1 20210110] at /usr/local/bin/python on Linux-5.4.226-129.415.amzn2.x86_64-x86_64-with-glibc2.31 [2023-01-12 073436,791] DEBUG {datahub.entrypoints:204} - GMS config {}

Ah looks like it didn’t make the release - entirely my bad. However, 0.9.6 does contain the fix - I’ve tested it locally myself (note that it prints out warnings a bit too much - I’m working on fixing that too 🙂 )

Thank you, I am able to delete now, but still some entries are missed

Discrepancy in entities yielded 0 and num entities 589. This means all entities may not have been deleted.

I hope you are working on this issue. Thanks for your help.

Yup we’re aware of that and are working on it. Just to make sure though, in the UI, everything that you expected to be deleted was actually deleted - is that right?

Yes, it deleted most, but left out few ( i think those related to that warning)

Great, we haven’t tried the fix yet, we’ll upgrade it soon. We did manage to delete all indices by adding a prefix to the ES indices and thus letting go of the old ones.

We did manage to delete all indices by adding a prefix to the ES indices and thus letting go of the old ones

Is it possible to share the commands? I would like to do the same

@careful-engine-38533 it’s a YAML change @better-fireman-33387 can you elaborate on what to change? https://datahubproject.io/docs/generated/ingestion/sources/elasticsearch/ Says you can add

url_prefix

but I am unsure about this

You need to use

global.elasticsearch.indexPrefix

in the helm chart values and set the prefix you like in order for all indices to share the same prefix

sorry to ask you again, what is the index value do i need to set for deleting the remaining dataset?

it’s just a prefix for all indices, since we couldn’t locate all indices names needs to be dropped we changed their names (it’s also much intuitive that all datahub indices will have the same prefix) thus forcing datahub to create new ones with your supplied prefix. it’s kind of a bypass