Hi, I created a connection MySQL > Bigquery. The thing is, it duplicated my entries by id, eventhough I entered the "full refresh / overwrite + deduplicate" option. My airbyte version is 2.1.0

Looking at your screenshots, you have a MySQL → BigQuery connection configured with Full Refresh | Overwrite + Deduped sync mode, but you're seeing duplicate rows for the same

id

in BigQuery. Here are a few things to check based on the Airbyte documentation: How Full Refresh - Overwrite + Deduped works In this mode, each sync replaces all data in the destination table and deduplicates the result using a primary key. The deduplication is determined by sorting the data using the cursor field and keeping only the latest row per primary key. [Full Refresh Overwrite+Deduped] Possible reasons for duplicates 1. Primary key not set correctly: The deduplication requires a primary key to be defined. If the

id

field is not configured as the primary key in your stream's schema/mappings, deduplication won't work as expected. Make sure

id

is set as the primary key for the

users

stream. [Sync Modes] 2. Sync resumption: The documentation notes that Full Refresh syncs may be resumed (e.g., to recover from transient errors), and when this happens, duplicates can occur. This is actually one of the reasons Overwrite + Deduped is recommended over plain Overwrite — to handle such cases. [Full Refresh Overwrite+Deduped] Recommended action • Go to the Schema tab of your connection and verify that

id

is explicitly set as the primary key for the

users

stream. • Also confirm that a cursor field (e.g.,

regtime

, which appears in your query) is configured, as deduplication sorts by the cursor field to keep the latest record. If the primary key and cursor field are correctly configured and you're still seeing duplicates, this may be a bug worth reporting to the Airbyte team with your version details (2.1.0). 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

there is no cursor field but yes id is primary key

Looking at your new screenshot, you selected "Replicate Source" during connection setup, which is the recommended option that maintains an up-to-date copy of your source data in the destination. [Select Sync Mode] This is consistent with the Full Refresh | Overwrite + Deduped mode shown in your earlier screenshot. Regarding the missing cursor field For Full Refresh - Overwrite + Deduped, the deduplication works by: • Using the primary key (your

id

field) to identify unique records • Using the cursor field to determine which version of a record is the most recent (keeping only the latest row per primary key) [Full Refresh Overwrite+Deduped] If no cursor field is configured, Airbyte may not be able to properly determine which duplicate row to keep, which could explain why both rows are being retained. What to try 1. Go to the Schema tab of your connection 2. Expand the

users

stream 3. Set a cursor field — based on your earlier query,

regtime

looks like a good candidate (it appears to be a registration timestamp) 4. Save your changes — you may need to trigger a Clear and re-sync to apply the changes cleanly [Select Sync Mode] Note that cursors and primary keys defined by the source connector are automatically selected and cannot be changed, but if they are not source-defined, you can set them manually in the Schema tab. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.