1. I tried to use I*ncremental Dedup | Append* mode at the beginning for stores table to replicate from Postgres to Snowflake but syncs failed with an error related to NULL values in the

_id

column. The source table contains rows where

_id

is NULL, and Airbyte's dedup/normalization step could not process these — causing the sync to abort. I checked Postgres stores table and there was no _id column with null values. Airbyte source (postgres) setting was using update method as 'scan changes with user defined cursor'. 2. Switched to Incremental | Append — now seeing duplicate rows: As a workaround we switched to Incremental|Append mode. Syncs now complete successfully, BUT we are seeing duplicate rows in Snowflake destination. What's unusual: all duplicate rows for a given store have the exact same

dateupdated

value. The only difference between duplicate rows is

_AIRBYTE_EXTRACTED_AT

— spaced ~15 minutes apart (our sync frequency). Example — 7 rows for same store in Snowflake, all identical data:

Idateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 12:46  ← first sync
dateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 13:03  ← duplicate
dateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 13:17  ← duplicate
dateupdated = 2026-07-14 12:33:17   _airbyte_extracted_at = 13:33  ← duplicate
... and so on

This only affects a small number of stores (~16 out of 243K) at any given time. The affected stores change over time. Question for community: What is causing Incremental|Append + UDC to replicate the same rows repeatedly with identical

dateupdated

? Is this a known behavior and is there a recommended fix short of switching to CDC?

Duplicate Rows with Incremental | Append + User-Defined Cursor Your two issues are related but distinct. Let me address each. Issue 1: Dedup Sync Failing with NULL _id There was a known bug in

destination-snowflake

versions 4.0.44 where NULL-safe primary key matching was accidentally removed from the dedup

MERGE

statement. This caused rows with NULL PK components to be re-inserted on every sync instead of being deduplicated, and could cause sync failures/unexpected behavior. [Snowflake dedup fix] The fix was released in destination-snowflake 4.0.45, which restores the NULL-safe form:

-- Restored NULL-safe PK matching:
(target.col = new.col OR (target.col IS NULL AND new.col IS NULL))

If you're on 4.0.44, upgrading the Snowflake destination connector to 4.0.45+ should resolve the dedup failure related to NULL values. [Snowflake dedup fix] Issue 2: Incremental | Append Producing Repeated Duplicates This is expected behavior given your setup, caused by two compounding factors: A. Inclusive cursor comparison Airbyte uses

cursor_field >= last_synced_cursor_value

(not strictly

>

). This means records sitting exactly on the cursor boundary are re-read on every sync. [Incremental Append - Inclusive Cursors] B. Cursor that doesn't advance When

dateupdated

is identical across all rows for a given store, the cursor value never moves forward. Every sync re-reads those rows and appends them again — which is exactly the pattern you're seeing (same

dateupdated

, different

_airbyte_extracted_at

spaced ~15 minutes apart). [Duplicate rows discussion] C. Postgres lookback window (additional factor) A recent fix to

source-postgres

added a 2-minute lookback window to timestamp cursors to avoid skipping records from in-flight transactions. This means even if the cursor did advance slightly, the connector re-reads the last ~2 minutes of previously-synced records on each sync. For append-only mode, this can produce some duplicates. [Postgres lookback fix] Why Only ~16 of 243K Stores? The affected stores are likely those whose

dateupdated

value happens to fall exactly on or near the cursor boundary at the time of each sync — meaning they were last updated recently and their timestamp hasn't changed since. As stores get new updates, they "escape" the boundary and new ones fall into it. Recommended Fixes The most pragmatic short-term fix without switching to CDC is to upgrade your Snowflake destination connector to 4.0.45+ and retry Incremental | Append + Deduped — the NULL PK issue that originally caused your dedup syncs to fail should now be resolved. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

how to upgrade this- upgrade your Snowflake destination connector to 4.0.45+