Hey all, does anyone know with incremental loads, ...
# replication-troubleshooting
z
Hey all, does anyone know with incremental loads, for the first sync does it retrieve all the data back to the start date? Then on subsequent runs it uses the
last_sync_max_cursor_field_value
, what happens if the previous sync failed for any reason? Does it take the last sync time of a successful run? https://docs.airbyte.com/understanding-airbyte/connections/incremental-append#known-limitations
✍️ 1
u
@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2748 to ensure timely resolution!
m
Airbyte works this way: • First sync, it won't have a state, so it will work as a full load basically. During the load and at the end it will emit state messages that will be saved to the connection. • Second sync and the following one, it will load the state from the connection and continue from that state onwards. The state usually have the cursor_field as main resource to get where to continue from. Now with the failure, I opened an issue for that, so I really don't know which is the correct behavior, as Airbyte team is still investigating, you can check it here: https://github.com/airbytehq/airbyte/issues/17774
In my opinion, airbyte should continue from where the last state emitted was, even it there was a failure. This will prevent large syncs to start from the beginning
h
Sure thanks a lot Marcelo for the help. @Zachary Damcevski let us know if you need any help
z
Thanks Marcelo, I'll keep an eye on that thread