Hello, I wanted to know what happens to the _scd t...
# replication-ask-ai
p
Hello, I wanted to know what happens to the _scd table in the destination v2. where can I see the history?
k
Hello, In Airbyte, when you use the Incremental Sync - Deduped History mode, two normalized destination tables are generated. One of these is a history table, also known as the _scd table (which refers to slowly changing dimension). This table includes historical versions of each record, including the start date and end date for each record [source](https://airbyte.com/tutorials/incremental-data-synchronization#what-is-the-difference-between-the-incremental-sync-modes). You can view the history table by executing a SQL command. For example, if your history table is called
table_two_scd
, you can view it by executing the following command:
Copy code
sql SELECT * FROM  table_two_scd;
[source](https://airbyte.com/tutorials/incremental-data-synchronization#exploring-incremental-deduped--history-synchronization--initial-creation-view-normalized-tables-in-the-destination-part-1) This will return a table that includes fields such as
_airbyte_unique_key
,
_airbyte_unique_key_scd
,
id
,
name
,
updated_at
,
_airbyte_start_at
,
_airbyte_end_at
,
_airbyte_active_row
,
_airbyte_ab_id
,
_airbyte_emitted_at
,
_airbyte_normalized_at
, and
_airbyte_table_two_hashid
[source](https://airbyte.com/tutorials/incremental-data-synchronization#exploring-incremental-deduped--history-synchronization--update-view-the-normalized-tables-in-the-destination-part-1). Please replace
table_two_scd
with the actual name of your history table.
p
@kapa.ai I want to know what happens in Destinations V2
k
Airbyte's Destinations V2 introduces several new features and improvements to enhance data integration: 1. One-to-one table mapping: In Destinations V2, data in one stream will always be mapped to one table in your data warehouse. This eliminates the creation of sub-tables ([source](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/)). 2. Improved error handling with
_airbyte_meta
: Airbyte will now populate typing errors in the
_airbyte_meta
column instead of failing your sync. This allows you to query these results to audit misformatted or unexpected data ([source](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/)). 3. Internal Airbyte tables in the
airbyte_internal
schema
: Airbyte will now generate all raw tables in the
airbyte_internal
schema. This means your destination schema will no longer be cluttered with raw data tables ([source](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/)). 4. Incremental delivery for large syncs: Data will be incrementally delivered to your final tables. This eliminates the need to wait hours to see the first rows in your destination table ([source](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/)). Destinations V2 was in preview for Snowflake and BigQuery during August 2023, and launched on August 29th, 2023. Other destinations will be transitioned to Destinations V2 on or before November 1st, 2023 ([source](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/)). Please note that the upgrade to Destinations V2 will require moving your connections to use updated versions of Airbyte destinations. Existing normalization options, both
Raw data (JSON)
and
Normalized tabular data
will be unsupported starting Nov 1, 2023 ([source](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2/)).
p
@kapa.ai how can I know if a record is deleted?
k
A message from kapa.ai
In Airbyte, the handling of deleted records depends on the type of data synchronization you are using. 1. Incremental Synchronization: In this mode, deleted records in the source are not propagated to the destination. This is because the records that are replicated to the destination are selected by a query on the source system for records that have been inserted or modified since the last synchronization. Any records that have been deleted are not returned by such a query, and therefore deleted records are not propagated to the destination ([source](https://airbyte.com/tutorials/incremental-data-synchronization)). 2. Change Data Capture (CDC) Synchronization: In this mode, deletions are handled differently. The raw data in the destination table will contain a new entry which indicates that a record has been deleted. The time of deletion is stored in _ab_cdc_deleted_at field ([source](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication)). For example, if a record with id=4 has been deleted, you would see an entry like this in the raw data:
Copy code
json {   "id": 4,   "_ab_cdc_deleted_at": "2022-09-26T18:26:08.629Z" }
The _ab_cdc_deleted_at field indicates the time the record was deleted. Please note that Airbyte is considering making a better detection of deletions in the source, especially with
Incremental
, and
Change Data Capture
based sync modes in the future ([source](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-append/)).