I used to work with Fivetran primarily (left because of price), so I'm trying to pick up more of the technicals of ELT now. Trying to get clearer on Incremental v. Incremental Append. My understanding is normal Incremental just syncs only new rows incrementally, usually decided by using a primary key, whereas Incremental Append syncs all rows in a short recent time period every sync, which creates duplicates in your warehouse. My understanding with something like Salesforce, because of their robust primary key, it usually wont need to be Incremental Append. Stitch for example doesnt append only this table. The documentation also just says Incremental, but there is still an option in the configuration to dedupe. Does a source like this actually end up needing to be deduped? https://docs.airbyte.io/integrations/sources/salesforce

Yes, because destinations and sources are decoupled In the sync modes, we have either for sources: • Incremental • full refresh And on destination, they follow the second word in the sync mode: • overwrite • append • dedupe history Then it’s just a combinaition of behavior of source with destination https://docs.airbyte.io/understanding-airbyte/connections

My understanding is normal Incremental just syncs only new rows incrementally, usually decided by using a primary key, whereas Incremental Append syncs all rows in a short recent time period every sync, which creates duplicates in your warehouse.

No, incremental in our case is always depending on a cursor field to retrieve rows in a short recent time period every sync We’ve introduced CDC that is closer to what you think of incremental: https://docs.airbyte.io/understanding-airbyte/cdc

Cool, thank you, I'm a lot clearer on this now.

I'm wondering if we can make the UI clearer. Something like "Source: Full refresh | Dest: overwrite" within the UI directly. Would that have made things clearer from the start?

Yes, I think that would be helpful. For me at least, knowing that first part was source related and second part was destination related did a lot to clarify.

Ok thx!

I created an issue here: https://github.com/airbytehq/airbyte/issues/3494