https://linen.dev logo
n

Nick Miller

01/20/2022, 6:01 PM
Hello! I’m trying to understand
Incremental - Append
. For typical SQL DBs it seems to reduce down to
SELECT * from table where cursor_column >= cursor_last_ran;
. I’ve previously built implementations with postgres similar to this, and one issue I’ve encountered with this that if there is a long running transaction/insert that the
SELECT *
may finish and update
cursor_last_ran
before the long running transaction finishes. A pretty common default would be a column called
updated_at
with a trigger/default value for
NOW()
. In postgres if you’re running inserts/updates, the 
updated_at
 column is updated when the row is updated. But that isn’t visible to the rest of the database until the transaction is committed, which could potentially be many seconds/minutes/hours later. The problem is that the 
updated_at
 value stays the same time after commit. Is this something Airbyte has an approach to handle?
Hello @Nick Miller the solution implemented right now should be generic enough to all use cases, of course there are some limitations as describe in our docs. Do you mind creating an issue in the Github project to discuss more about this topic?
g

Guido Turturici

01/21/2022, 11:57 PM
thanks!
2 Views