Yusuf Mirkar
05/17/2023, 9:58 AMkapa.ai
05/17/2023, 9:58 AMkapa.ai
05/17/2023, 9:59 AMYusuf Mirkar
05/17/2023, 10:03 AMkapa.ai
05/17/2023, 10:03 AMkapa.ai
05/17/2023, 10:04 AMYusuf Mirkar
05/17/2023, 10:04 AMkapa.ai
05/17/2023, 10:04 AMkapa.ai
05/17/2023, 10:06 AMincrement|deduped history
replication mode and had normalization enabled. The destination tables had the following row counts: - raw_table: 2.5 million rows - table: 2.1 million rows - table_scd: 2.6 million rows The user expected the scd
and raw
tables to have the same number of rows. However, the support team explained that: - raw
tables append all records (new ones and old ones) - table_scd
stores the history of all records (new ones, old, and deleted ones too) - table
is the final table with deduplicated records The difference in row counts between the raw
and scd
tables could be due to the fact that the scd
table also stores deleted records, while the raw
table only appends new and old records. The main table has deduplicated records, which is why it has a lower row count. It's important to note that the row counts in the destination tables can vary depending on the replication mode, normalization settings, and the nature of the data being synced.Yusuf Mirkar
05/17/2023, 10:08 AMkapa.ai
05/17/2023, 10:08 AMkapa.ai
05/17/2023, 10:09 AMincrement|deduped history
replication mode, which still doesn't capture deleted rows. If you need to capture deleted rows, you should consider using [Change Data Capture (CDC)](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication) replication. CDC replication captures modifications on the source database, including deletions, and transmits them to the destination database. This is different from standard incremental replication, which cannot return deleted records and does not correctly transmit deletions from the source to the destination. For more information on CDC replication, refer to this [tutorial](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication).Yusuf Mirkar
05/17/2023, 10:11 AMincremental | deduped history
and not cdc
so how did he got deleted rows in table_scd ?kapa.ai
05/17/2023, 10:11 AMkapa.ai
05/17/2023, 10:12 AMYusuf Mirkar
05/17/2023, 10:15 AMincremental | deduped history
there will be two tables in destination - table_scd and final table, right ?
table_scd will have all new and old records but not deleted ones as it is not cdc
and final table will have only deduped records, right ?kapa.ai
05/17/2023, 10:15 AMkapa.ai
05/17/2023, 10:16 AMYusuf Mirkar
05/17/2023, 10:18 AMkapa.ai
05/17/2023, 10:18 AMkapa.ai
05/17/2023, 10:18 AM