Hi, when I turned on Incremental Sync + Deduped + History on a table, I set the cursor field to be

updated_at

-- when I manually ran a sync, I see an identical row for each primary key now. The

updated_at

is the same for both, and all the other fields are the same except the

_AIRBYTE_UNIQUE_ID

column where one contains an id and the other is

NULL

. This was unexpected, as I expected it to dedupe. The first sync I did the mode was Full Refresh + Overwrite, then I switched it to Incremental Sync + Dedupe + History, but I reset the connection so it should have cleared the table. Any ideas on what I'm missing here?

Hi. It is very difficult to say exactly what is the issue since you didn't specify the source. updated_at is not enough as a primary key since it is a one to one function. there could be same created_at with the same updated_at and it is not a reverse function since it could be null. you need more keys to make this work. null is not a primary key since it will never be the same address in memory nor same value. you need (depending on the data) transaction_id, customer_id etc.. some thing more unique for each row so that the merging will be able to say that if the composite keys are the same then overwrite the values and if not the same add new row. right now it says they are never the same.

@Marcos Marx (Airbyte) turned this thread into Zendesk ticket 2432 to ensure timely resolution!

Hello Eli. Thanks for the response. I don't have access to your ZenDesk to see the ticket. The table does have a primary key. The source is PostgreSQL, and the PKID is

BIGINT

. When I changed the sync mode, I specified

created_at

as the cursor field, and airbyte auto-magically detected

id

as the PKID.

Hey @Davis Ford, thanks for the question. Did you make any changes to the underlying schema? Maybe this may explain the limitations of incremental + deduped + history. https://docs.airbyte.com/understanding-airbyte/connections/incremental-deduped-history/#known-limitations

@Davis Ford I am running into the exact same issue. Any resolutions that you've found?

@Annika Maybin sadly, I did not. There were a number of technical limitations with AirByte that caused us to move to a different solution for ELT.

@Davis Ford Bummer. Thanks for your answer.