A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

I’m testing stateful ingestion. Is it a correct assumption that datasets with `removed=true` for the status aspect does not appear in the UI?

I’m checking and the _removal event_ is correctly generated
```RecordEnvelope(record=MetadataChangeProposalWrapper(entityType='dataset', changeType='UPSERT', entityUrn='urn:li:dataset:(urn:li:dataPlatform:redshift@bigbrain,dwh_sch.ba.houston_experiments_means,PROD)', entityKeyAspect=None, auditHeader=None, aspectName='status', aspect=StatusClass({'removed': True}), systemMetadata=None), metadata={'workunit_id': 'soft-delete-table-urn:li:dataset:(urn:li:dataPlatform:redshift,dwh_sch.ba.houston_experiments_means,PROD)'})```
However, the UI is still serving the metadata for the dataset
```<https://datacatalog.datahighway-dev.schip.io/dataset/urn:li:dataset:(urn:li:dataPlatform:redshift@bigbrain,dwh_sch.ba.houston_experiments_means,PROD)/Schema?is_lineage_mode=false>```
:thinking_face:

Hmm: I think if you specifically bookmark the entity url and go to it, you will see it. But the entity should no longer be reachable from "search", "browse" etc.

Yes that is true! It should be hidden from all means of _discovery_

You are true!
Direct access with the URN still retrieves the dataset (I guess this is expected behaviour then).
However, I cannot find it with the search.

i'd say it's not ideal - we should ideally post a label at the top of the page saying "this dataset has been removed" in red :slightly_smiling_face:

Screenshot 2022-01-27 at 20.43.56.png

Screenshot 2022-01-27 at 20.43.26.png

Well, now that I’m testing it more… I have some inconsistent results.
The one I deleted is the one with platform `redshift@bigbrain`
It does not appear in the first search :ok_hand:
But it does in the second :warning:

Note the search term in the second is `houston_experiments_means`

Screenshot 2022-01-27 at 20.46.49.png

And more surprisingly, it is not shown when I click on _Explore all…_ from the suggestion in the search bar. Even that I’m using the same search term `houston_experiments_means` :thinking_face:

<@U01NN9CTQ01> Are we only applying isRemoved filter to some things?

<@U027ZS25RFS> How recently did you remove this? We do have a 10minute query cache

So if you issued the same search before, it might be used the cached results :stuck_out_tongue:

dinner time here, so I will check later :wink:

+1 I think it may be due to caching. TTL is 10 minutes

Yes. Now I get consistent results: no search returns the removed one. Thanks for the support!

&gt; i’d say it’s not ideal - we should ideally post a label at the top of the page saying “this dataset has been removed” in red
+1 on this, this’d be a nice feature