2. INCREMENTAL (our first version of incremental is an append only approach -- so if a record is updated, we add a new copy of the record in the destination, but we do not do anything to delete the old one, the user is expected to handle de-duping using built-in functionality in the warehouse or dbt) - The approach for transferring data in this case is similar to previous case. We only send the new / updated data from the source to the destination. It is all written into a temporary table. If that is completely successful, within a transaction the temporary table is copied into the main table. In this case, by design, it is highly likely that you will end up with multiple copies of different versions of the same record in the destination. The contents of each delta should still have exactly once delivery and should never be missing records.

We will be adding support for doing the de-duping mentioned in my incremental explanation as well, but it will not be part of the first release of incremental.

1. That is correct. On the roadmap for subsequent release but not in our first release of incremental.

2. Not quite! All of the destinations we have written so far leverage transactions in the target to handle making operations atomic. When we write the S3 integration we should be handle writing data in atomic fashion. If you have specific features in mind for S3, lmk and we will take them into consideration. It will just be a little different implementation than say most relational databases which all have somewhat similar transaction abstractions.

3. I am not very familiar with Pulsar. For Kafka, I think we would be shooting for at least once delivery of every message, but I'm not sure we can do exactly once. So there might be duplicates.

we are still figuring out the priority of it. we've definitely had a bunch of people mention it.

is this the main blocker for airbyte being useful to you? if so, would you be willing to share a bit more about your use case? the more info we have on why this blocks people from getting us from it, the higher we'll prioritize it.

i think this could potentially be something we start working on in january (but please don't hold me to that yet). and i think we have some clear ideas on how we would do it.

i'll make a node to post in this thread again or in general when we figure out when we are doing this.

scroll down to the bottom.

our roadmap is in our docs https://docs.airbyte.io/roadmap. it's a little vague on this point right now, but will add more clarity on it there once we have it.