BTW, for the incremental sync, airbyte requires to...
# advice-data-ingestion
y
BTW, for the incremental sync, airbyte requires to define a cursor over destination’s target table. Suppose my destination is a postgreSQL table (called “abc”) with the column name “start_time” as cursor. Could someone point me to the code / query that airbyte will get the highest value of “start_time” in each incremental sync? Is it running some query like
select max(start_time) from abc
?
✅ 1
a
Hi @Yudian, the database connector perform this query to get rows more recent than the last cursor value stored in Airbyte state:
SELECT * FROM your_table WHERE cursor_column > last_cursor_value
y
thank you @Augustin Lafanechere (Airbyte), could you elaborate more on how to get the
last_cursor_value
? Will airbyte store a state for the
last_cursor_value
, when each time it performs a sync? Something like
last_cursor_value
= max of cursor value of data in last incremental sync? Did airbyte store
last_cursor_value
in some places?
a
Yes exactly, Airbyte checkpoints the cursor value in a state object which is stored in the Airbyte database.
👍 1