Hi Team, I'm testing right now Airbyte for databa...
# ask-community-for-troubleshooting
k
Hi Team, I'm testing right now Airbyte for database replication with my
MariaDB <> BigQuery
setup and I have a question regarding sync mode. I have configured binlog for my MariaDB database to thrive from CDC functionality and now wondering what's the best/proper sync mode option for me as I cannot find much enough information in docs. Some of my tables have rows that are updated in the meantime and as far as I know, whether I give
source sync mode
to Incremental or Overwrite, both of them will capture this change, but what about destination mode? Is
append
option will update my row/cell where the change has happened or it will append an entire row and I'll end up with duplicate rows? And what's the reason that my setup doesn't let me to use
Deduped history
option?
1
m
With CDC always use incremental dedup history
If Airbyte can allow you use dedup is because your table doesn’t have primary key which doesn’t allow receive the correct updates from the cdc
k
@Marcos Marx (Airbyte) That is actually impossible, because all of my tables have PK, especially the most typical table 'users'. Is there any chance that CDC is not configured correctly and doesn't recognize PK in my tables?
Ok, nvmd. I found out that it doesn't work when I choose destination - BigQuery (denormalized typed struct), but when I user regular - BigQuery it shows me deduped option 🙂
👍 1