Nick Miller
01/20/2022, 6:01 PMIncremental - Append . For typical SQL DBs it seems to reduce down to SELECT * from table where cursor_column >= cursor_last_ran; . I’ve previously built implementations with postgres similar to this, and one issue I’ve encountered with this that if there is a long running transaction/insert that the SELECT * may finish and update cursor_last_ran before the long running transaction finishes.
A pretty common default would be a column called updated_at with a trigger/default value for NOW(). In postgres if you’re running inserts/updates, the updated_at column is updated when the row is updated. But that isn’t visible to the rest of the database until the transaction is committed, which could potentially be many seconds/minutes/hours later. The problem is that the updated_at value stays the same time after commit. Is this something Airbyte has an approach to handle?Nick Miller
01/20/2022, 9:27 PMNick Miller
01/20/2022, 11:22 PMGuido Turturici
01/21/2022, 11:57 PM