Hi! I would like to share with you a possible issue we have found when combining stateful ingestion and transformers. This is how you can reproduce it and the evidences we have found along the way: • Set up an ingestion recipe with stateful ingestion enabled for the source + any transform (eg. add_dataset_ownership transform) • Run the recipe adding some deny pattern (just to force the soft deletion of some dataset). At this point we have checked the logs and the generated events look totally correct with two events for the affected dataset: one from the soft-removal with

Status(removed=True)

and the second event for the

Ownerhip

aspect (nothing about the status in the second upsert). • The dataset is wrongly shown in the UI as a valid dataset (not soft-deleted). We have also checked the backend and the dataset has

Status(removed=False)

. So if the issue is not during the ingestion, it must be the backend the one deciding to enable back the dataset for some reason. Looking for something supporting our assumption we have found this in the source code https://github.com/datahub-project/datahub/blob/8c8f1b987a0c9fc29f4005aa8d132ad2550f3f05/metadata-io/src/main/java/com/linkedin/metadata/entity/EntityService.java#L1097 I could be wrong but it looks like in some cases, the backend decides to set the removal flag to false. It’s like it decides to re-enable back the dataset because there are other aspects being updated. If that’s true and while it could make sense in some cases, it causes our simple use case to misbehave. WDYT? Could be that the root cause of the issue?

Sure, will take a look

My team is having this issue as well, hence we wanted to use entitiesv2 endpoint to check for urn existence and status before insertion. But this endpoint will always return a result despite providing a urn that doesnt exist. It will always provide a datasetKey minmally.

Hi @dazzling-judge-80093, is there any update on this? The use of transforms sets back the status removed flag to false so in practice disables stateful ingestion feature. I assume use of transforms is very common so this would be impacting many users.

Sorry, I’m busy some other stuff but it is definitely on my list to check

Just adding more insights in order to be helpful 😄 We have noted that deprecating functions in the transformers were introduced in this PR https://github.com/datahub-project/datahub/pull/4337 It could be that the new support for MCP in the transformers introduced some side-effect. Adding @loud-island-88694 to the loop since he was the author of that PR.

Thanks for flagging this @witty-butcher-82399, we're taking a look

ok, I think I know what is the problem. Due to legacy reasons, we convert MCP to fake mce, and because of this, the transformer generates ownership for the delete aspects which basically reactivates the node.

So the problem is exclusively with the ownership transform?

it doesn’t set back

Here? https://github.com/datahub-project/datahub/blob/8c8f1b987a0c9fc29f4005aa8d132ad2550f3f05/metadata-io/src/main/java/com/linkedin/metadata/entity/EntityService.java#L1097

This is what I can see:

{
    "auditHeader": null,
    "entityType": "dataset",
    "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:bigquery,myproject.partition_test.users,PROD)",
    "entityKeyAspect": null,
    "changeType": "UPSERT",
    "aspectName": "status",
    "aspect": {
        "value": "{\"removed\": true}",
        "contentType": "application/json"
    },
    "systemMetadata": {
        "lastObserved": 1656583428073,
        "runId": "bigquery-2022_06_30-12_03_31",
        "registryName": null,
        "registryVersion": null,
        "properties": null
    }
},
{
    "auditHeader": null,
    "entityType": "dataset",
    "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:bigquery,myproject.partition_test.users,PROD)",
    "entityKeyAspect": null,
    "changeType": "UPSERT",
    "aspectName": "ownership",
    "aspect": {
        "value": "{\"owners\": [{\"owner\": \"urn:li:corpuser:username1\", \"type\": \"PRODUCER\"}, {\"owner\": \"urn:li:corpGroup:groupname\", \"type\": \"PRODUCER\"}], \"lastModified\": {\"time\": 0, \"actor\": \"urn:li:corpuser:unknown\"}}",
        "contentType": "application/json"
    },
    "systemMetadata": {
        "lastObserved": 1656583428073,
        "runId": "bigquery-2022_06_30-12_03_31",
        "registryName": null,
        "registryVersion": null,
        "properties": null
    }
},

So, what’s the conclusion so far? • The problem is exclusively with the ownership transform • Or the problem is with the transforms using the fake mce function (so all dataset transforms?) • Or the problem is with all the transforms And more important: is there an ETA for a fix? or still soon for that?

The issue is more with those transformers which can generate additional aspect(s) and not just change a specific aspect. I’m thinking/working on a workaround.

OK, I see. thankyou1

like owner transform can add owner even if there were no owner before

Adding an aspect such as owner shouldn’t be a problem, or no more problem than updating. IMO the problem is when transforms add them in different events: status aspect and owner aspect arrive separately to the backend and so backend decides to fill in the second event with the status(removed=false) aspect.

Hi @witty-butcher-82399! I believe this has been addressed in this PR & will be included in our next release! We’re a bit short-staffed this week due to some much-deserved PTO, but we should have one out within the next couple of weeks!

Thanks Maggie for sharing the update and the PR 👍