Adrian Bakula
02/02/2023, 6:09 PM0.4.46 (these changes). Specifically with this:
FULL_REFRESH will see partial data if fails part way through, this is being addressed in phase 2 with a table in destination that tracks when a sync starts and ends
1. How can we tell on our side whether a full refresh was successful or not? We can have downstream dependencies that read from the raw data tables, but without checking via airbyte API whether the sync was successful for the connection that carries that table, there's no way for us to be sure. In cases where you have an orchestration tool this is fine, but there are cases where you read table contents without orchestration.
2. Relatedly, if we have tables of the form Full Refresh | Append , are we just going to have bad data in our raw tables forever? I suppose this is fine, but it puts deduplication burden on our end to make sure we aren't ingesting the same data twice because we had a one time failure.
3. I found the communication on this to be poor. The changelog on the connector website here just says
0.4.46 2023-01-26 #20631 Added support for destination checkpointing with staging
even though potential breaking changes were identified in the pull request and explicitly mentioned. This can have huge user impact and should be communicated explicitly. I happened to find it since I was digging through your blog post about release notes, but that shouldn't be where I find out about this kind of thing.
Thanks all 😄user
03/14/2023, 2:57 PMAdrian Bakula
03/24/2023, 9:11 PMAdrian Bakula
05/05/2023, 4:08 PM