Nick Miller
01/20/2022, 6:01 PMIncremental - Append
. For typical SQL DBs it seems to reduce down to SELECT * from table where cursor_column >= cursor_last_ran;
. I’ve previously built implementations with postgres similar to this, and one issue I’ve encountered with this that if there is a long running transaction/insert that the SELECT *
may finish and update cursor_last_ran
before the long running transaction finishes.
A pretty common default would be a column called updated_at
with a trigger/default value for NOW()
. In postgres if you’re running inserts/updates, the updated_at
column is updated when the row is updated. But that isn’t visible to the rest of the database until the transaction is committed, which could potentially be many seconds/minutes/hours later. The problem is that the updated_at
value stays the same time after commit. Is this something Airbyte has an approach to handle?Guido Turturici
01/21/2022, 11:57 PM