@kapa.ai Hey, i wonder which connection mode i should use, Incremental Sync - Append + Deduped or Incremental Sync - Append. My goal is to update data within my destination in ways of slowly changing dimension type2. Is my understanding correct that deduped will delete old entries of the same primary key and create a new entry with the updated data, while simple append creates a new row for each entry newer then the cursor ?

Your understanding is partially correct, but let's clarify the details based on the provided knowledge sources. ### Incremental Sync - Append - Behavior: This mode results in a final normalized table that contains a historical copy of every record that has been inserted or updated on the source and synced to the destination. If a record has been updated and synchronized multiple times, it will result in multiple entries in the final normalized table. - Use Case: This is useful if you want to keep a complete history of all changes, as it appends new rows for each update. - Reference: [Incremental Sync - Append](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append) ### Incremental Sync - Append + Deduped - Behavior: This mode updates rows that have been modified instead of adding a new version of the row with the updated data. It combines the rows in the destination and uses the most recent data. The final table will be unique per primary key, meaning old entries of the same primary key will be replaced with the new data. - Use Case: This is useful if you want to maintain only the latest version of each record, ensuring no duplicates based on the primary key. - Reference: [Incremental Sync - Append + Deduped](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped) ### Slowly Changing Dimension Type 2 (SCD Type 2) - Behavior: SCD Type 2 involves keeping a full history of changes in the data warehouse. This means that each change results in a new row, preserving the historical data. - Relevant Mode: The

Incremental Sync - Append

mode aligns more closely with SCD Type 2 because it keeps a historical copy of every record, including all changes. ### Conclusion - Incremental Sync - Append: This mode is more suitable for your goal of implementing Slowly Changing Dimension Type 2 (SCD Type 2), as it creates a new row for each entry newer than the cursor, preserving the history of changes. - Incremental Sync - Append + Deduped: This mode will not meet your requirement for SCD Type 2, as it de-duplicates and keeps only the latest version of each record, effectively deleting old entries of the same primary key. For more detailed information, you can refer to the following links: - [Incremental Sync - Append](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append) - [Incremental Sync - Append + Deduped](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped) If you have any further questions or need additional clarification, feel free to ask!