An option to truncate the final destination table ...
# give-feedback
c
An option to truncate the final destination table instead of recreating it in the case of
Full Refresh - Overwrite
e
Sorry - not at this time. Especially as we are trying to build syncs with 0 data downtime, we sync data to a new table and then at the end of the sync swap the old and new tables into place. This is both faster and cheaper then running a
delete
on the existing data too.
c
0 data downtime
- can’t this be done in a transaction?
e
Where we can, yep! The swapping of the new and old final tables is done in a transaction... although some destinations (bigquery) don't offer transactional guarantees on metadata operations, of which a rename is.
c
instead of a swap - what about a truncate and insert? I don’t think I understand the difference, and in some destinations recreating the final table keeps us from using things like timetravel in Snowflake
e
It could, and in the future we might optimize this, but for the new table + rename lets us handle other use-cases as well (e.g. schema migrations and type changes) with a bit more resiliency. We have not prioritized switching to truncate+insert at this time.
Just to double check - the snowflake docs do seem to indicate that time travel does work across renames - https://docs.snowflake.com/en/user-guide/data-time-travel#example-dropping-and-restoring-a-table-multiple-times Are you seeing something different?
c
🤔 - I’ll have to test that out ; in the past, we’ve had to undrop tables to get the timetravel