Hi Everyone, Trying to use https://datahubproject.io/docs/metadata-ingestion/transformers/#mark-dataset-status But don't see the change reflect (the dataset is still visible in the UI)

{
  "source": {
    "type": "glue",
    "config": {
      "env": "PROD",
      "aws_region": "us-east-1",
      "extract_transforms": "false",
      "table_pattern": {
        "allow": [
          "testdb.*$"
        ],
        "ignoreCase": "false"
      }
    }
  },
  "transformers": [
    {
      "type": "mark_dataset_status",
      "config": {
        "removed": "true"
      }
    }
  ]
}

Hi Keethi, do the logs show it getting caught at all? Is it having an issue finding the dataset or just with the transformer execution?

@orange-night-91387 - No errors -

[2022-06-23 16:45:56,816] {logging_mixin.py:104} INFO -  Pipeline config is {'sink': {'type': 'datahub-rest', 'config': {'server': '<https://hostname>'}}, 'source': {'type': 'glue', 'config': {'env': 'PROD', 'aws_region': 'us-east-1', 'extract_transforms': 'false', 'table_pattern': {'allow': ['information_schema.*'], 'ignoreCase': 'false'}}}, 'transformers': [{'type': 'mark_dataset_status', 'config': {'removed': 'true'}}]}
[2022-06-23 16:47:53,632] {base.py:78} INFO - Using connection to: id: datahub_rest_default. Host: <https://hostname>, Port: None, Schema: , Login: , Password: None, extra: None
[2022-06-23 16:47:53,636] {base.py:78} INFO - Using connection to: id: datahub_rest_default. Host: <https://hostname>, Port: None, Schema: , Login: , Password: None, extra: None
[2022-06-23 16:47:53,734] {_lineage_core.py:67} INFO - Emitted from Lineage: DataFlow(

Created issue here: https://github.com/datahub-project/datahub/issues/5242

ok . Thank you !