1. I tried to use I*ncremental Dedup | Append* mode at the beginning for stores table to replicate from Postgres to Snowflake but syncs failed with an error related to NULL values in the

_id

column. The source table contains rows where

_id

is NULL, and Airbyte's dedup/normalization step could not process these — causing the sync to abort. I checked Postgres stores table and there was no _id column with null values. Airbyte source (postgres) setting was using update method as 'scan changes with user defined cursor'. 2. Switched to Incremental | Append — now seeing duplicate rows: As a workaround we switched to Incremental|Append mode. Syncs now complete successfully, BUT we are seeing duplicate rows in Snowflake destination. What's unusual: all duplicate rows for a given store have the exact same

dateupdated

value. The only difference between duplicate rows is

_AIRBYTE_EXTRACTED_AT

— spaced ~15 minutes apart (our sync frequency). Example — 7 rows for same store in Snowflake, all identical data:

Idateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 12:46  ← first sync
dateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 13:03  ← duplicate
dateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 13:17  ← duplicate
dateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 13:33  ← duplicate
... and so on

This only affects a small number of stores (~16 out of 243K) at any given time. The affected stores change over time. Question for community: What is causing Incremental|Append + UDC to replicate the same rows repeatedly with identical

dateupdated

? Is this a known behavior and is there a recommended fix short of switching to CDC? Screenshot 2026-07-15 154107.jpg \

Issue 1: NULL _id Error with Incremental Append + Deduped There is a known bug in

destination-snowflake

versions starting at 4.0.44 where NULL-safe primary key matching was accidentally removed from the dedup

MERGE

clause. This caused rows with NULL PK components to be re-inserted on every sync instead of being deduplicated, and could cause sync failures. A fix was released in destination-snowflake 4.0.45 that restores the NULL-safe comparison (

target.col = new.col OR (target.col IS NULL AND new.col IS NULL)

). [Snowflake dedup fix] Recommendation: Update your Snowflake destination connector to 4.0.45 or later and retry

Incremental | Append + Deduped

with a proper primary key configured. Issue 2: Duplicate Rows with Incremental | Append This is expected behavior, and here's why: Root Cause: Inclusive Cursor + Non-Advancing Cursor Value Airbyte uses an inclusive cursor comparison:

SELECT * FROM table WHERE cursor_field >= 'last_sync_max_cursor_field_value'

This means records where

dateupdated

equals the last synced cursor value are re-read on every sync. This is intentional — it guarantees Airbyte doesn't miss records written at the exact same timestamp as the last sync. [Inclusive Cursors] For the ~16 affected stores, their

dateupdated

value is not advancing — it stays the same across syncs. So the cursor never moves past those rows, and every sync re-reads and re-appends them. [Incremental Append] Additionally,

Incremental | Append

never deduplicates — it only ever adds rows to the destination. [Incremental Append docs] Recommended Fixes Option 1: Switch to Incremental | Append + Deduped (Recommended) Once you update the Snowflake destination connector to ≥4.0.45 (fixing the NULL PK issue), switch back to

Incremental | Append + Deduped

with a proper primary key. This will keep only the latest version of each record per primary key. [Append + Deduped] Option 2: Use a Better Cursor Field Ensure your cursor column (

dateupdated

) actually advances on every row update. If some rows have a static

dateupdated

that never changes, the cursor will never progress past them. A monotonically increasing column (e.g., a real

updated_at

that changes on every write, or an auto-incrementing ID) works best. Option 3: Deduplicate Downstream in Snowflake If you want to keep

Incremental | Append

for the raw history but need a clean latest-state view, you can deduplicate in Snowflake:

SELECT *
FROM your_table
QUALIFY ROW_NUMBER() OVER (PARTITION BY <primary_key> ORDER BY dateupdated DESC, _airbyte_extracted_at DESC) = 1

[Discussion] Summary: The duplicate rows are caused by the inclusive cursor re-reading rows whose

dateupdated

hasn't changed, combined with Append mode never deduplicating. The original dedup failure was likely the Snowflake NULL-safe PK bug (fixed in 4.0.45). Upgrading the connector and switching back to

Append + Deduped

with a proper PK is the cleanest solution. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

will Snowflake destination connector to ≥4.0.45 break exsiting conections or any issues?