From my testing, Airbyte does not continue an ingestion if something went wrong in the first attempt, even though the state was saved and is registered on the connection. From what I investigated, this is due to Airbyte only loading the state when the job starts, and using that state for all attempts, even though records were flushed to the destination database, making so that on any failure, Airbyte just loads everything he just did again (x3 because of default number of attempts). I opened an issue for that https://github.com/airbytehq/airbyte/issues/17774. Bizarrely enough, if after the first job fails all attempts, and I run a second job, it will continue from where it left of, making so that it is better to have 1 attempt configured, that way the state is respected and the load will not be duplicated unnecessarily. My question is, is this by design? Can we make so that the attempt loads the current state from the connection and continue from where it stopped on the first one? Even if this is by desing, for me there is a bug here, because the second job should ignore the saved state from the first attempt and try to start the sync from the beginning, just like the attempts do. It is just not consistent what AIrbyte is doing now For me this is a show stopper, because we have very big syncs, and they can fail a lot, making that loading dozens of GB every attempt for no gain at all.

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2716 to ensure timely resolution!

Hi! Thanks for opening up that issue. I see a discussion has already started in the comments there, so let's continue it on GitHub.

Hi! Thanks for opening up that issue. I see a discussion has already started in the comments there, so let's continue it on GitHub.

Hey Nataly! Can you drive some attention to that issue internally? I really do think this is a big issue and may affect the usability of Airbyte as a whole. I may be over reacting but I don't think it is a light problem.