:wave: Hi there! I wanted to raise a concern with ...
# replication-troubleshooting
a
👋 Hi there! I wanted to raise a concern with one of the more recent updates to the Snowflake Destination Connector, specifically version
0.4.46
(these changes). Specifically with this:
Copy code
FULL_REFRESH will see partial data if fails part way through, this is being addressed in phase 2 with a table in destination that tracks when a sync starts and ends
1. How can we tell on our side whether a full refresh was successful or not? We can have downstream dependencies that read from the raw data tables, but without checking via airbyte API whether the sync was successful for the connection that carries that table, there's no way for us to be sure. In cases where you have an orchestration tool this is fine, but there are cases where you read table contents without orchestration. 2. Relatedly, if we have tables of the form
Full Refresh | Append
, are we just going to have bad data in our raw tables forever? I suppose this is fine, but it puts deduplication burden on our end to make sure we aren't ingesting the same data twice because we had a one time failure. 3. I found the communication on this to be poor. The changelog on the connector website here just says
Copy code
0.4.46	2023-01-26	#20631	Added support for destination checkpointing with staging
even though potential breaking changes were identified in the pull request and explicitly mentioned. This can have huge user impact and should be communicated explicitly. I happened to find it since I was digging through your blog post about release notes, but that shouldn't be where I find out about this kind of thing. Thanks all 😄
u
Hello Adrian Bakula, it's been a while without an update from us. Are you still having problems or did you find a solution?
a
@Saj Dider (Airbyte) not really. To be quite honest with you, I find that there is very little visibility in the changes that happen in the connectors we care about, and what I found here makes me terrified of upgrading. I haven't followed along with hesitation I have here, because I don't really know where I should be looking. Outside of reading every PR and comments, there doesn't seem to be good communication here, and these are mission-critical tools for us at this point. So I'm not entirely sure where to look, who to follow up with, and where I can voice my concerns on these things. Any feedback here?
Hey all, just wanted to know if anyone had any further opinions on this. We just had an incident on our end because of an upgrade to the snowflake destination connector (exactly the concern that I brought up), and it was very unclear that there was a breaking change in this upgrade (this ended up being updated by another team member). This is an understandably hard problem to solve, but who can we talk to to see if we can improve the communication and visibility on the Airbyte side on breaking changes for connectors going forward?