we're looking into triggering a workflow based on ...
# replication-troubleshooting
r
we're looking into triggering a workflow based on whether Salesforce data that we sync into Snowflake has changed. should i assume that if we set up the SFDC->Snowflake sync to incremental deduped, it should use/support CDC (and thus support tracking using table streams)?
u
Airbyte doesn't have the option to listen sources and trigger a connection when new data has arrived. You need to select a specific option cron/sync frequency to run and retrive new records.
r
yep, i've got that. my question was more about how the data is loaded into destination tables with each sync
with a full overwrite, i imagine a CDC log would see "all rows were deleted" + "all these new rows were just added", even if none of the underlying data changed at all (which would prevent us from using CDC to track changes, i imagine)
my question is – if we set up the sync as incremental/append, will the data changes in the destination table happen in a way that Snowflake's CDC can accurately track row changes?
u
Append only new add new records without taking care of deleted ones. You must select the incremental append + dedup to always clean the deleted records.
r
sorry, Marcos, maybe i'm not explaining myself properly. i just want to know if incremental append + dedup will play nice with Snowflake's CDC
u
CDC is related to source connectors not destination, let's say if you want to retrieve records from Postgres with soft-deletes and updates using CDC will reflect in your final table.
r
this is what i'm trying to use: https://docs.snowflake.com/en/user-guide/streams.html#change-tracking-using-table-streams if a row in a Postgres-to-Snowflake table that is syncing in incremental append + deduped mode changes from
id: 1; value:'abc'
to
id: 1; value:'def'
, will Snowflake's CDC be able to tell that the row with
id: 1
was updated, or will it view it as "this row was deleted; this row was added"?
u
No, it will only work if you use Postgres CDC feature which captures the metadata from the source and them you need to use Snowflake destination with incremental + dedup to update the destination tables. The final result will what you're expecting.