i am currently using incremental | append + deduped for all of my syncs. However, my ideal solution would be to use "full refresh | overwrite". I don't use the full refresh because I am trying to save on network costs. is there a way to use "incremental | append + deduped" for most of the syncs, and then perhaps once per week I can do a "full refresh | overwrite"?

Yes, you can achieve this by configuring your syncs to use "Incremental | Append + Deduped" for most of the time and then manually triggering a "Full Refresh | Overwrite" sync periodically, such as once per week. Here’s how you can do it: 1. Set Up Incremental Sync: - Configure your syncs to use the "Incremental | Append + Deduped" mode. This mode will help you save on network costs by only syncing the changes since the last sync. 2. Manual Full Refresh Sync: - Once a week, you can manually trigger a "Full Refresh | Overwrite" sync. This will ensure that your data is fully refreshed and any discrepancies are corrected. ### Steps to Configure 1. Configure Incremental Sync: - Go to your Airbyte dashboard. - Select the connection you want to configure. - Set the sync mode to "Incremental | Append + Deduped". 2. Manual Full Refresh Sync: - On the day you want to perform a full refresh, go to the same connection. - Change the sync mode to "Full Refresh | Overwrite". - Trigger the sync manually. - After the sync is complete, revert the sync mode back to "Incremental | Append + Deduped". ### Known Limitations Be aware of the known limitations of the "Incremental | Append + Deduped" mode. For instance, if modifications to the underlying records are made without properly updating the cursor field, the updated records won't be picked up by the incremental sync as expected. This is because the source connectors extract delta rows using a SQL query like:

sql select * from table where cursor_field > 'last_sync_max_cursor_field_value'

[Source](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations) ### Additional Resources - [Incremental - Append + Deduped](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append-deduped) - [Full Refresh - Overwrite](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-overwrite/) By following these steps, you can optimize your sync strategy to balance between network cost savings and data accuracy.

@kapa.ai two follow-ups. [1] can i do this automatically instead of manually every week? [2] can i do this for every table in a particular connection? doing this for every single table will take forever, i want to do it for all tables

Based on the information provided in the knowledge sources, here are the answers to your follow-up questions: ### [1] Can I do this automatically instead of manually every week? Currently, there is no direct mention in the provided knowledge sources about automating the detection and addition of new tables without manual intervention. According to a forum post, users have to manually refresh to detect new tables: > "is there any way to automatically detect and then add the new tables, instead of click refresh manually? ... it means we need refresh everyday?" ([Source](https://discuss.airbyte.io/t/2315)) ### [2] Can I do this for every table in a particular connection? Doing this for every single table will take forever, I want to do it for all tables. There is no direct method mentioned in the knowledge sources to apply changes to all tables in a connection automatically. However, you can filter the tables you want to sync to reduce the time: > "You can filter only the tables you want to reduce the time." ([Source](https://discuss.airbyte.io/t/3731)) Additionally, there is a known issue where changing the schema setting causes previously selected tables to be lost, which can be time-consuming if you have many tables: > "When selecting tables to sync for a connection you would expect that if you needed to make changes to the schema later that the tables would persist and still be selected when using the 'Update latest source schema' option. But after clicking this button ALL tables are then selected by default." ([Source](https://github.com/airbytehq/airbyte/issues/4295)) ### Conclusion - Automatic Detection: Currently, there is no automated way to detect and add new tables without manual refresh. - Applying Changes to All Tables: There is no direct method to apply changes to all tables in a connection automatically. You may need to manually select or filter tables to manage them efficiently. For more detailed and updated information, you might want to refer to the [Airbyte documentation](https://docs.airbyte.com/) or reach out to the Airbyte community for potential workarounds or upcoming features.