Hey folks, just onboarding to the tool. Love how s...
# ask-community-for-troubleshooting
g
Hey folks, just onboarding to the tool. Love how simple it is to get started. Can you confirm if this is best practice? 1. I have an orders table that gets updated if it changes, no updated_at field unfortunately. Should I use
full refresh - overwrite
, since I need the latest and greatest, and the primary key will not be enough as a cursor field? 2. I have an order_carts table that gets updated if there's a new event, with a created_at field. Existing rows will never be changed. In this case, I think I can use
incremental - append
, since it would just add the new fields since the last sync.
1
d
That seems right!
If the source supported CDC, for #1 you can also use CDC with incremental append dedup. A
deleted_at
column will appear in the destination table. Queries can use this column as a filter to know what is deleted. There is some complexity but the benefit is not always having to sync all the data. Not necessary if the original table is small
g
Thanks Davin! If choose CDC, will it be more efficient, since it can just look at what's been changed since the last sync? That's my main use case - dump replica to warehouse run my transformations. I skimmed the docs a few times but this is new to me.
d
Yes. Incremental does that via a primary key. CDC goes on step further and is also able to detect deletes.
g
I see. I was worried because I ran the full refresh yesterday and it was in the magnitude of hours, but incremental updates should be a lot speedier (mins/secs) now. Thanks again.
airbyte rocket 1
@Davin Chia (Airbyte) Last question - did a miss a checkbox? I don't see any table constraints in my destination location. Or does that have to be done manually by me?
d
What do you mean by table constraints?
g
I had some primary and foreign key constraints on my table (source), but I don't see any once it was migrated over
d
Ah yes we only replicate data - constraints aren't moved over. Is this a blocker?