Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hey, about *idempotent job* I looked in Github issues and didn't find anything about it. Generally DE jobs try to be idempotent, that is, if they are turned again with the same parameters they should have the same result. Today there is a simple trigger in the UI and the API just receives the `connectionId` to trigger the job ... is there anything about it under discussion?

Source jobs take in a few inputs (explicitly or implicitly):
1. The structure of the data (the catalog)
2. Sync-related configuration (like full refresh vs incremental pulling of records)
3. For incremental, some state object, which usually contains pointers to the last data read.
4. The current datetime 
5. The current data in the source API/db

Since a lot of the data we read can change in the underlying API/db it’s difficult to say that syncing a connection (even if you fixed 1-3) should result in the same output.

For syncs configured as full refreshes, the behavior is idempotent (% timestamps/metadata) and the output is the same assuming the underlying source API/db did not change.

We’re also planning on working on a “snapshot view” of data where it isn’t just incremental-append but gives you a “latest” view of the data, which sounds close to what you’re hoping for (if I understand correctly)?