Hi everyone, how can I used airbyte to handle all updated records from source database. Scenario: I have three records in my source db and I have loaded it to my destination db at the first loading time. Before the second loading time, a record was updated in the source-db but that record still has the same primary_id after update. I noticed that this updated record was added as a new record in the destination db, so I now have two record for the same primary_id. Please, how can i solve issues like this?

Hi @Ariyo Kabir it really depends on what strategy you want to have to handle these update: If you want your destination to mirror the source database you can choose the

Full refresh - overwrite

sync mode. If you want to keep the state of each field you need to have a column which saves the timestamp at which the update was made and use this column as a cursor in an

incremental - append

sync mode. If you source supports CDC you can also decide to make use of CDC to capture all the state of your record. I encourage you to read this documentation for a better understanding of the sync modes we offer.

Thanks for your response Augustin. Firstly, its not optimal enough to be moving all the table every 30 minutes, because my database keeps increasing day-in day-out. Secondly, on the changing of my date column to string datatype, I strongly believe that using dbt could be useful, but I don't have the luxury time to start learning dbt now, I don't if you can be of help on how use dbt to change some source datatype to another datatype in the destination db

Here @Ariyo Kabir our docs about dbt+normalization: https://docs.airbyte.com/operator-guides/transformation-and-normalization