Hello everyone,
I have a connection from Postgres ...
# replication-troubleshootingb
Bruno Agresta González
12/23/2022, 1:34 PMHello everyone,
I have a connection from Postgres (AWS) to BigQuery. In this connection I am experiencing problems with some tables that have “Incremental Deduped + History” as sync configuration. Cursor Field “Updated_at” and Primary key “id”.
The problem is that in the destination table the rows that are updates appear duplicated. This is not the behavior I expect for the deduped configuration. I’m using the
0.40.241.2.90.3.26m
Marcos Marx (Airbyte)
12/23/2022, 3:58 PMAre the dedup records in first-level/parent tables? Can you share a sample of limited view?
a
Adam Bloom
12/23/2022, 4:55 PMincremental + deduped for postgres does not deduplicate correctly without normalization enabled. Are you loading raw json or with normalization?
Adam Bloom
12/23/2022, 4:55 PM(I agree that this behavior isn't documented anywhere, btw - just what I observed when I tested this many months ago)
Adam Bloom
12/23/2022, 4:56 PMMy solution was to run deduplication within our dbt model that is loading from the raw data
Adam Bloom
12/23/2022, 4:57 PMAre you using CDC syncs? That's the sync mode that we're using, and that I'd expect this to occur in. CDC can have multiple updates for the same row and they are not consolidated in the WAL by design. Airbyte itself has no logic/capability to detect and handle this.
b
Bruno Agresta González
12/26/2022, 2:35 PM• I’m not using Change Data Capture.
• Running deduplication within dbt it’s an option. I think the only option.
• I’m using normalization configuration.
• The dedup records are in the parent tables. I attached some screenshot.
m
Marcos Marx (Airbyte)
12/27/2022, 6:24 PMWhat field did you use for cursor field and primary key?
b
Bruno Agresta González
12/28/2022, 9:10 PMI used: Cursor Field “`updated_at`” and Primary key “`id`”.
3 Views