Why does the full refresh -overwrite keeps creating duplicates by extracted_at? If there is just one entry for each "id_bo" in the original table

Looking at your screenshots, you have a single entry for

id_bo = 4193

in your Google Sheet, but the BigQuery table shows 3 rows with different

_airbyte_extracted_at

timestamps. This is expected behavior in Airbyte, and here's why: Why duplicates appear with Full Refresh - Overwrite Airbyte is an at-least-once delivery system. Even within a single sync, Airbyte paginates through the source data in chunks. If a sync fails partway through and needs to retry, records from the already-processed chunk will be re-sent on the next attempt, resulting in duplicate rows with different

_airbyte_extracted_at

timestamps. [Resumability] Additionally, each time a Full Refresh - Overwrite sync runs successfully, it replaces all data in the destination. The multiple timestamps you see suggest the record was written across multiple sync runs or retry attempts. [Full Refresh - Overwrite] How to fix this If avoiding duplicates is a strong requirement, Airbyte recommends switching to Full Refresh - Overwrite + Deduped instead of plain Overwrite. This mode: • Still replaces all data each sync • But also deduplicates the final table by primary key, keeping only the latest row per unique key > "If avoiding duplicate is a strong requirement, we recommend using Overwrite + Deduped over Overwrite. Bear in mind that deduplication incurs additional warehouse costs." [Full Refresh - Overwrite + Deduped] In your case, setting

id_bo

as the primary key with the Overwrite + Deduped mode would ensure only one row per

id_bo

exists in the final BigQuery table. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai is it new this behaviour? because I think i was using this method before and it didn't create duplicates

The knowledge sources don't provide a clear timeline of when this at-least-once delivery behavior was introduced. However, based on the documentation, this is not a new behavior — Airbyte has always been described as an at-least-once delivery system, and resumability (which can cause duplicates within a sync) applies to all sync modes. [Resumability] What may have changed is that Resumable Full Refresh became more explicitly supported as of Airbyte v1.0.0, which could mean retries and re-delivery of records became more common or more visible after that version. [Resumability] It's also possible that in previous syncs, no retries were triggered (e.g., no network issues or failures), so duplicates simply didn't occur in practice — even though the system always had the potential to produce them. In short, the capability to produce duplicates has always existed by design, but you may not have encountered it before if your earlier syncs completed without any retry attempts. The safest way to guarantee no duplicates in your final table is to use Full Refresh - Overwrite + Deduped. [Full Refresh - Overwrite + Deduped] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai why does it run without ending? there are all the data already (ex : additional_data_adgroup=146/146)

Looking at your screenshot, the sync has been running for 11+ minutes with streams either actively syncing or queued, even though some streams appear to have extracted all their records (e.g.,

additional_data_adgroup

shows 146 extracted). This is actually a known issue with the destination-bigquery connector. Based on the GitHub issue in the knowledge sources, syncs can hang indefinitely even after all data has been flushed. The logs in that issue show: • Queue size drops to 0 records / 0 bytes • Workers report "nothing to do" • But the sync never terminates on its own The root cause appears to be related to how the BigQuery destination handles the final flush and termination signal. In some cases, a

null

job value is returned by BigQuery (

Received null value for Job, nothing to waitFor

), and the sync gets stuck waiting for a terminal status that never arrives. [sync hangs forever] A similar issue was also reported where a sync completes all data loading but fails because no terminal stream status message is ever received, causing the sync to hang or fail with a

TransientErrorException

. [terminal status issue] What you can do: • Cancel and retry the sync — in many reported cases, the data was actually loaded correctly despite the hang • Check your BigQuery tables to confirm the data is already there before retrying • Report the issue to Airbyte if it happens consistently, as both linked GitHub issues are still open The knowledge sources do not provide a definitive fix for this behavior yet, as these are open bugs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.