Hello guys, I've set an ingestion pipeline

snowflake-beta

with stateful ingestion and it was working just fine. Then a new version was released and

snowflake-beta

doesn't exist anymore, it was replaced by

snowflake

. However, all my statefull ingestion set with

snowflake-beta

doesn't work anymore. I've now old tables that were already dropped from snowflake but which remains on the Catalog after running an ingestion with

snowflake

and

stateful ingestion enabled

. Is there a way to solve this problem ? Thanks !

cc @hundreds-photographer-13496 - I was under the impression that the state setup would be compatible between snowflake-beta and snowflake, so this definitely shouldn’t be happening

@salmon-angle-92685 can you please confirm if deleted tables you are noticing have only Queries/Stats and no Schema ? There is a known issue that we are addressing at the moment. I'm trying to understand if the issue you are facing is related to it or something different.

@hundreds-photographer-13496 Nope, it has schema. It has everything just like a non-deleted table. :/

okay. can you confirm if same

pipeline_name

was used for both

snowflake-beta

and then

snowflake

recipe ? From my understanding, as far as pipeline_name is same across

snowflake

and

snowflake-beta

sources, stateful delete should work as expected.

Any updates ? @hundreds-photographer-13496 🙂 Thank you so much !

Hey @salmon-angle-92685 I didn't get time to reproduce it. The temporary solution for you to fix it would be to soft delete all snowflake datasets for once and then re-ingest using snowflake ingestion:

datahub delete --soft --platform snowflake --entity_type dataset

However I am very much interested in understanding why it went wrong in your case in the first place and will get back when I get chance to reproduce this next week.

@hundreds-photographer-13496 The problem of doing your approach is that I'll lose all the metadata. I asked all the developers of the company to add their metadata and now I don't want to ask them that again 😕 Would you have another temporary solution for me ? Thanks :)

Hey @salmon-angle-92685 soft delete will not remove metadata, it'll just hide (soft-delete) all the tables. When the recipe runs again, the tables (that really exist and are not deleted) will come to life again along with all the metadata that you would have added manually.

@hundreds-photographer-13496 Aaaaah, okay ! But, won't it keep "trash data" on the storage ? I've always asked myself that, when I read the docs about soft-delete. Thanks for all your support btw !

you are right .