Hi <@U04SE163WC8> does airbyte 0.59.1 not create s...
# ask-ai
n
Hi @kapa.ai does airbyte 0.59.1 not create scd table anymore ?
k
Yes, you are correct. Starting from a certain version, Airbyte has stopped creating SCD (Slowly Changing Dimension) tables. The reasons for this decision include: 1. The SCD tables were found to be inaccurate. They only sampled data when a sync happened, missing intermediate changes between syncs. This was especially the case for sources where CDC (Change Data Capture) was not available. 2. The SCD tables were rarely used. The majority of Airbyte users only used the final normalized tables. 3. The creation of SCD tables added cost and slowed down syncs. The cost of normalization in your warehouse was at least doubled by the creation of SCD tables in addition to the final tables. However, you can recreate similar SCD tables by running dbt at the end of a sync and creating 'snapshot' tables. Please refer to this [Github discussion](https://github.com/airbytehq/airbyte/discussions/31681) and this [Github issue](https://github.com/airbytehq/airbyte/issues/25194) for more details.
n
so if we choose incremental append how we know the current data @kapa.ai?
k
A message from kapa.ai
When you choose the Incremental Append sync mode in Airbyte, the current data is determined by the cursor field. Some sources are able to determine the cursor that they use without any user input. For example, in the exchange rates source, the source knows that the date field should be used to determine the last record that was synced. In these cases, you simply select the incremental option in the UI ([source](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append#source-defined-cursor)). However, with Incremental Append, a record which was updated in the source will be appended to the destination rather than updated in-place. This means that if data in the source uses a primary key (e.g:
user_id
in the
users
table), then the destination will end up having multiple records with the same primary key value. If you only care about having the latest snapshot of your data, you may want to look at other sync modes that will keep smaller copies of the replicated data or you can periodically run cleanup jobs which retain only the latest instance of each record ([source](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append#getting-the-latest-snapshot-of-data)). To view the content of the incremental append synchronization into the raw table, you can execute the following SQL command:
Copy code
sql SELECT * FROM _airbyte_raw_table_one;
This should respond with a table that contains the data from the incremental append synchronization ([source](https://airbyte.com/tutorials/incremental-data-synchronization#step-6)).