when i am building lineage, it seems to me that if the dataset does not exist, we create an empty dataset and then build the lineage. Is it possible to optionally turn off this feature and just fail lineage building instead?

Could you create a feature-request for this? This approach makes sure lineage ingestion is robust even when different sources are writing different parts of the lineage. Are you expecting the ingest request to fail when any URN references do not exist?

yes i would expect lineage to fail in this case if the url references do not exist. Is there a way to do this at the moment?

So the reason this happens is because sometimes you actually ingest lineage before getting around to ingesting the sources tables. for example ingesting airflow lineage before ingesting your Snowflake instance. We don't necessarily want to force an order of ingestion

hmm, but is there a way to validate if a dataset exists or not?

^ This happens to us as well. We

getDataset

in GMS where the dataset doesnt exist, but it still returns something. From the code I see that number of aspects are checked. If only

key

returns, then it is treated as non-exist.

can you please share an example? I would like to have this mechanism to keep the datasets ingestion robust

Please refer to method

tryProvisionUser

, there is a line try to get if CorpUser exists. It checks if returned Snapshot has more than 1 aspects

Yes- currently the backend (GMS) is programmed to return a basic Key aspect when provided an URN. This permits cases where links in the Graph can be created before the entity themselves are populated. For example, Airflow lineage between 2 Snowflake datasets is emitted before Snowflake ingestion is run. Whether this is optimal in the long term is unclear. Would it be useful to have an additional endpoint for checking whether an entity exists (has had aspects generated for it) or not? That's something we could definitely do

absolutely that would be helpful since then consumers can optionally use it.

Thanks - let me add a feature request and we can try to get to it shortly. Quick question - how do you intend to use the feature? Will you be building monitoring on top?

I would actually just monitor the dataset if it exists before pushing lineage, and if it doesnt lineage doesnt get built.