@kapa.ai does deduplication on the sync mode increase the time the sync takes

Yes, deduplication in sync modes does increase the time a sync takes. Here's why: Deduplication Adds Processing Overhead How it works: In modes like Incremental Append + Deduped or Full Refresh Overwrite + Deduped, Airbyte must sort records by cursor field and identify the latest row per primary key, removing older duplicates. This extra processing step takes time and compute resources. [Sync Modes Docs] Warehouse compute costs: Typing and deduplication can increase your destination's compute costs, which scale with the amount of data being transformed. [Typing & Deduping] Real-World Examples of Slowdowns • Full table scans: With CDC Incremental Dedup on a large Postgres → Redshift sync, the normalization phase scanned the entire SCD table (500M rows) on every sync, even when there were no changes — taking hours per sync. [Forum: CDC Dedup] • Query planner degradation: A change in the Redshift destination connector (v4.0.4) caused dedup JOIN predicates to fall back from hash joins to nested loops, turning a 2-minute sync into a 17–43+ minute sync. [GitHub Issue] • ClickHouse behavior: For ClickHouse specifically, the dedup cost is paid asynchronously by background merges (so syncs themselves are faster), whereas most other destinations pay the dedup cost synchronously during each sync. [ClickHouse Docs] Summary The trade-off is that deduplication gives you a clean, unique-per-primary-key final table, but at the cost of additional sync time and warehouse compute. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements