Hi all. After spending some time here in the weekend i finally build my first connector (source). Super good experience and i am excited to see the future of airbyte 🙂 However is there any particular reason why it is not possible to do scd on full_refresh tables? I mean in some cases it might be necessary to track hard deletes or maybe the source do not have a way to do incremental extract? Or is this determined by the destination?

Hello! Glad it went smoothly and you enjoyed it!

However is there any particular reason why it is not possible to do scd on full_refresh tables?

do you mean you would like to have an option to do

full_refresh + dedup history

to generate the

_scd

tables ?

@Chris (deprecated profile) Yes exactly and then maybe a column like "__airbyte_deleted_row"_ which can be true or false. When you do full_refresh you will be able to check for hard deletes, which might occur in some source systems.

Could you create a new github issue to track this new feature request: I briefly mentionned it in the past but we focused on getting simple deduplication on incremental runs out first… https://github.com/airbytehq/airbyte/issues/2371#issuecomment-795247997 This new destination sync mode would involve more transformation work to figure out that some records were dropped and flag them as

_airbyte_deleted_row