if airbyte sync starts at :00 and done at :20 then...
# replication-ask-ai
y
if airbyte sync starts at :00 and done at :20 then does data betweem :00 and :20 get synced?
k
Yes. For an incremental sync, Airbyte uses a cursor (for example an
updated_at
timestamp) and on each run it only pulls records whose cursor value is greater than the last stored cursor value from the previous sync. [Incremental syncs; Tutorial incremental] So if: • A sync starts at
:00
and finishes at
:20
, and • The cursor for that sync ends up at some time within that
:00–:20
window, then all records with cursor values up to that final cursor are considered synced. On the next sync, Airbyte will query only records with cursor values greater than that stored cursor, i.e., changes that happened after what was already captured. [Tutorial incremental] The docs do not specify behavior in terms of exact wall-clock minutes like
:00
and
:20
, only in terms of cursor values and state; but conceptually, data whose cursor falls between the previous cursor and the new cursor (which may correspond to
:00–:20
) is what gets synced in that run. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai so if a sync happens every hour that takes 20 min to run, that mean it's syncing data between xx:20 - xx:20 essentially
k
Not exactly. The docs only define incremental syncs in terms of cursor values and state, not wall‑clock windows like
xx:20–xx:20
. [Incremental syncs; Tutorial incremental] What is guaranteed: • Each run stores a cursor (e.g., max
updated_at
) in state when it finishes. [Incremental syncs] • The next run only pulls records whose cursor value is greater than (or greater than/equal to, depending on mode) that stored cursor. [Tutorial incremental] So if a sync runs every hour and takes ~20 minutes: • Run N: reads “everything with cursor > last_cursor”, finishes, and writes a new
last_cursor_N
. • Run N+1: reads “everything with cursor > last_cursor_N”. The effective window is between the previous run’s stored cursor and the current run’s stored cursor, which usually correlates with, but is not defined as,
xx:20–xx:20
on the clock. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai is this the same behavior with full refreshes?
k
No. Full refresh behaves differently from incremental syncs. For full refresh (both append and overwrite variants): • Each sync run always reads all available data from the source, regardless of what was read in previous runs. [Full refresh append; Full refresh overwrite] • There is no cursor-based “since last time” window like with incremental sync. So if a full refresh runs every hour and takes 20 minutes, each run re-reads the entire dataset, not just data between the previous and current run times. The difference between variants is only what happens in the destination: • Full refresh | overwrite: destination table is fully replaced with the new copy. [Full refresh overwrite] • Full refresh | append: new full copy is appended to existing data, creating duplicates over time. [Full refresh append] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.