https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> does overwrite for a destination ov...
# ask-ai
m
@kapa.ai does overwrite for a destination overwrite only the synced records for a given date range / run or does it overwrite the entire table?
k
The Overwrite Sync mode in Airbyte overwrites the entire table in the destination, not just the synced records for a given date range or run. This means that all previously synced data in the destination table will be deleted and replaced with the new data from the source. This is part of the Full Refresh Overwrite synchronization mode, which always retrieves all available data requested from the source and writes it in the destination, regardless of whether the data has previously been synced or not. One of the advantages of this approach is that data that has been deleted from the source will also be deleted in the destination. However, it's important to note that this may not be possible for all destinations, and in some cases, the existing data may need to be erased at the start of each full-refresh sync. Sources: - [Starburst Galaxy destination user guide > Features](https://docs.airbyte.com/integrations/destinations/starburst-galaxy#features) - [Destination-specific mechanism for full refresh](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/full-refresh-overwrite#destination-specific-mechanism-for-full-refresh) - [Database synchronization with full refresh overwrite](https://airbyte.com/tutorials/full-data-synchronization#step-7) - [What is full refresh and when to use it Part 1](https://airbyte.com/tutorials/full-data-synchronization#step-2)
m
@kapa.ai is there any concept of upsert where you may want to update a record based on a set of keyed columns?
k
A message from kapa.ai
Yes, Airbyte does support the concept of upserts through its Incremental Sync - Append + Deduped mode. In this mode, the delta from a sync will be appended to the existing history data in the data warehouse. In addition, it will update the associated record in the final table based on a set of keyed columns, which are typically the primary keys of the table. Here's an example from the Airbyte documentation: Assume that 
updated_at
is our 
cursor_field
and 
name
is the 
primary_key
. Let's say the following data already exists in our data warehouse. | name | deceased | updated_at | | --- | --- | --- | | Louis XVI | false | 1754 | | Marie Antoinette | false | 1755 | In the next sync, the delta contains the following record: | name | deceased | updated_at | | --- | --- | --- | | Louis XVI | false | 1785 | At the end of this incremental sync, the data warehouse would now contain: | name | deceased | updated_at | | --- | --- | --- | | Marie Antoinette | false | 1755 | | Louis XVI | false | 1785 | As you can see, the record for Louis XVI was updated (upserted) based on the 
name
primary key. However, it's important to note that due to the use of a cursor column, if modifications to the underlying records are made without properly updating the cursor field, then the updated records won't be picked up by the Incremental sync as expected. Sources: - [Incremental Sync - Append + Deduped > Rules](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#rules) - [Incremental Sync - Append + Deduped > Known Limitations](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations)
Open in Slack
PreviousNext
Powered by