Hey team, I wanted to start a discussion about how datasets get represented over time. In particular I'm thinking of a situation where a dataset is created, ingested, edited in Datahub, and deleted at some point. Then some time in the future, a new dataset with the same name and in the same database/schema is created. The same urn would be generated in both these cases, so internally they would be treated as the same dataset. And if the original dataset was soft-deleted, then the subsequent one would inherit that custom documentation even if it doesn't apply to this new case. Should these be considered two separate datasets? Would the second one be considered a new version of the original? If the original dataset had it's status set to removed when it was deleted from the database, would there be some indication that it had been removed and then reingested? I think there is a lot to consider here. I'm interested to hear what Datahub and the community's perspective is on this scenario, both from a stateful and stateless ingestion standpoint, and if there is a particular direction Datahub has in mind. Thanks!

The current behavior would be that the soft-deleted flag would get flipped back to false, and hence the new table would inherit data from the previous one, and there wouldn’t be any indication that it had been removed and reingested.

Thanks for the reply Harshal. This is a feasible solution, but not an preferred or practical one imo. There's a couple issue I see: 1. Scalability: You can do the hard delete easily when we have a specific dataset in mind, but what about if you have a process that regularly ingesting databases with tens of thousands of datasets each? 2. Historical data: By hard deleting you lose the historical data of that original dataset. It may or may not be useful to keep that historical data on hand.

In general, we do try to preserve version history - we actually retain a history of every edit made to every entity. When a dataset is soft-deleted, no data is actually deleted - we just stop surfacing that dataset in search/browse, but you can still access it via a direct URL as you would any other dataset.

I appreciate the input. I think the last point is a solid one and something we were considering. And you're right, I think implementing that as a separate blanket rule with it's own timetable would be much more manageable than determining what stays and what goes at ingestion time.

Yep that’s correct!