Hi, I am having an issue where duplicates are occurring during a Full Refresh | Overwrite sync. • Deployment method: OSS version 0.40.18 on EC2 • Connection details: ◦ MySQL (connector version 10.0.13) -> Redshift (connector version 0.3.51) ◦ All tables set to Full Refresh | Overwrite ◦ Runs 1 time a day on a CRON Schedule • Error info: ◦ Only in a handful of tables, am I seeing duplicate records in the destination table. ◦ Here is an example where the destination table has a row count of 162 786 rows ◦ When I look at the logs for that specific table, I get the following info about the stream. And this is the same patter for the other tables with the duplicates. What could be causing the duplicates? Both rows have the same Airbyte emitted at datetime stamp:

{
    "streamName" : "anonymous_table",
    "stats" : {
      "recordsEmitted" : 81394,
      "bytesEmitted" : 16462411,
      "recordsCommitted" : 81394
    }

Hey, was this working normally for you before / on a different version? Also are you using staging for Redshift? If you haven't already could you create a github issue about this?

Hello Frank Kody, it's been a while without an update from us. Are you still having problems or did you find a solution?

Jinx @Marcos Marx (Airbyte) 😆

Hi! Yes I resolved the issue. It was related to my discourse post: https://discuss.airbyte.io/t/redshift-destination-buffer-stream-uploading-duplicate-files-to-s3-staging-area/3313

Thanks for sharing your solution!