Hey hey! Looks like the snowflake destination connector always sets the table retention policy to 0 days on table creation. Any reason for this? Would it make sense to make this configurable or at least allow to not have this line in case we want the retention policy to just follow the schema level retention? Thanks 😄

Airbyte uses 0 days to disable the time travel and reduce costs. https://docs.snowflake.com/en/sql-reference/parameters.html#data-retention-time-in-days Number of days for which Snowflake retains historical data for performing Time Travel actions (SELECT, CLONE, UNDROP) on the object. A value of 0 effectively disables Time Travel for the specified database, schema, or table. For more information, see Understanding & Using Time Travel. I didn't find any other issues related to the Time Travel options in Airbyte Github.

Gotcha! For context: We have a handful of cases where we have just-in-time availability snapshots of data that we are storing in Snowflake using

Incremental - Append

and

Full Refresh - Append

to have history. However, the connections allow you to

Reset

the data via UI and API and there don't seem to be any restrictions on that (nor can you configure restrictions on this). We have concerns on our end that we will lose the history we have built up, causing permanent and unacceptable data loss. One solution to this is to set up time travel in snowflake so that we can recover the data in the event of a random reset. We were hoping to do 1 or more days so that we have a recovery option in case of data loss. Additionally, time travel as you might imagine might be handy in other cases as well (maybe a developer iterates on a custom connector and we have a dev error that writes faulty data that we need to rewind).

Airbyte uses 0 days to disable the time travel and reduce costs.

The data in our case is not very large. Meaning there is very little storage cost impact on our side. So the cost saving measure, while thoughtful, is far less important to us than having the potential for data recovery. Happy to put in a github issue for this.

I totally agree this can be a configurable parameter in destination. Can you create a Github issue for that?

Yep! Created here

Thanks I request to connector team take a look in it!