Hi, We are using open source airbyte and running on linux. Full Refresh Overwrite sync is not doing full overwrite of table, instead we see data get duplicated. Source is SQL Server with no CDC enabled. There is no specific error in log indicate any issue . In status, it show accurate row count of source being loaded but in destination, row count is significantly higher than that indicating that data is not getting overwritten but added. Has anyone faced this issue?

What is the destination connector? What version of Airbyte and the source/destination connectors?

airbyte 2.1.0, source - sql server Microsoft SQL Server 2016 (SP3) (KB5003279) - 13.0.6300.2 (X64) and destination is snowflake. snowflake authenication is key based and snowflake User used has access to DDL and DML. Source connector MSSQL 4.4.8 , destination connector snowflake version - 4.0.43

It's possible that this is related to resumable full refresh where the connection is getting interrupted and starting over. In this case duplicate rows could exist in the sync, but will all get written due to the overwrite type. If your data has a primary key of some type, you might want to try the

Full Refresh | Overwrite + Deduped

option if it's available, as it would at least prevent the effects of this issue. I don't use the Snowflake destination (only the source), but it's possible that there's something weird with the

SWAP WITH

that gets run at the end of an overwrite sync (to take the temporary table and overwrite the normal one). You might also double-check that you don't have a duplicate copy of your Airbyte sync running on the same schedule, as this could cause messy results.

almost 90% of table don't have primary keys that is why we want to keep it full refresh overwrite. we did any duplicate jobs.

we saw this with Postgres when destination tables had constraints and dependencies like materialised views created from them, so airbyte seems to fail to drop/recreate destination and sync just flows into existing table on top of the existing Data

Note that I resolved the issues by truncating the tables and resyncing.