can anyone point me to a `Incremental Sync - Dedup...
# ask-community-for-troubleshootingj
Jarrod Parkes
08/31/2021, 6:49 PMcan anyone point me to a
Incremental Sync - Deduped Historyj
Jeff Crooks
08/31/2021, 6:56 PMthere be dragons there
š 1
Jeff Crooks
08/31/2021, 6:56 PMits an incremental sync, but then actually a full refresh to dedupe
Jeff Crooks
08/31/2021, 6:57 PMIMO its a little bloated to accomplish that based on my use cases, so I handle with my own dbt normalization
āļø 1
j
Jarrod Parkes
08/31/2021, 7:01 PMi was under the impression that you either incremental or full, but not both at the same time š¤ ?
a
abhi
08/31/2021, 7:04 PMHey @Jarrod Parkes, have you checked out the doc for it? If so, let me know what was insufficient about it, would love to help clear it up š
j
Jarrod Parkes
08/31/2021, 7:04 PMi have, i got it open right now
Jarrod Parkes
08/31/2021, 7:05 PMi have the pieces, im pretty sure ā a primary key per record and a cursor field per record. im just trying to figure out how to connect the dots
Jarrod Parkes
08/31/2021, 7:05 PMi have my current custom source working with a full refresh (tested append/overwrite), but im trying to transition it to this increment sync dedupāed as ive learned more about what airbyte can do and what my use case requires
a
abhi
08/31/2021, 7:07 PMAh, do you need an example on writing the Incremental + dedupe implementation?
j
Jarrod Parkes
08/31/2021, 7:08 PMthat is what im looking for help with yes. the doc you linked doesnāt really show/discuss how to define a catalog/stream for this use case ā that is where im lost
Jarrod Parkes
08/31/2021, 7:08 PMive tried to find one by searching the repo, and stumbled across the BambooHR source, but the way its stream is defined seems pretty custom
a
abhi
08/31/2021, 7:10 PMhttps://github.com/airbytehq/airbyte/blob/master/docs/connector-development/tutorials/adding-incremental-sync.md
I'm not sure that this covers deduplication, but this covers the basics of getting incremental sync running
j
Jarrod Parkes
08/31/2021, 7:10 PMwill check out
a
abhi
08/31/2021, 7:15 PMIf you're using the CDK, the Shopify connector might be a better example: https://github.com/airbytehq/airbyte/blob/abde7c77278ce6efb221a37c82a71a629d7d3b2e[ā¦]integrations/connectors/source-shopify/source_shopify/source.py
j
Jarrod Parkes
08/31/2021, 7:29 PM@abhi it looks like shopify example is capturing state on a per-stream basis. i think what i wanted to do was to capture this on a per-record basis?
Jarrod Parkes
08/31/2021, 7:32 PMbecause each individual record that is returned by this API has its own ācursor fieldā
Jarrod Parkes
08/31/2021, 7:48 PMor it the case that you store one updated_at value representing the last time a sync was run and then you can filter across individual records by using that value
Jarrod Parkes
08/31/2021, 7:49 PMseems like that is what is happening here https://github.com/airbytehq/airbyte/blob/abde7c77278ce6efb221a37c82a71a629d7d3b2e[ā¦]integrations/connectors/source-shopify/source_shopify/source.py
j
Jeff Crooks
08/31/2021, 8:31 PM@Jarrod Parkes I believe the cursor is set per stream, is that what you are asking?
Jeff Crooks
08/31/2021, 8:31 PMIE, its not a blanket value for all streams, but based on whatever the last value was for each particular stream
j
Jarrod Parkes
08/31/2021, 8:43 PMyeh, per stream is what im seeing too
Jarrod Parkes
08/31/2021, 8:43 PMi was more trying to understand how i use that to effect what gets sent for the next incremental sync
j
Jeff Crooks
08/31/2021, 8:55 PMit will pull that state and get everything >= cursor on the next exec
10 Views