https://linen.dev logo
Join Slack
Powered by
Hello all, Can we configure to run a stream only f...
# help-connector-development
m
Hello all, Can we configure to run a stream only for incremental sync using low code cdk.
k
A message from kapa.ai
Yes, you can configure a stream for incremental sync using the low-code CDK. To do this, you need to specify a 
DatetimeBasedCursor
on your stream's 
incremental_sync
field. This will allow your stream to be configured so that only data that has changed since the prior sync is replicated to a destination. More information on 
incremental_sync
configurations and the 
DatetimeBasedCursor
component can be found in the [incremental syncs](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/incremental-syncs) section of the documentation.
j
This isn't possible.
In general, the first sync of a connection (or after a reset) will always be a full refresh
What's your use case?
Do you want to prevent users from accidentally configuring an expensive connection?
m
I am trying to sync Quickbooks Invoices. I am using alpha connector of the quickbooks to sync the invoice stream. But this stream will not give me deleted records information. So I am trying to use quickbooks change data capture api for invoices in Quickbooks connector. But cdc returns many records as we sync from the start.
j
By definition the first sync will do a "full refresh" as the destination doesn't have any data yet, so there is no "incremental" update
A lot of connectors work around this by having a "start date" configuration that defines the point in time from where the connector should start syncing
m
oh!! okay. Can you please guide me on how to configure a start date as current time - 1 day ?
j
That depends on the API, in case of quickbooks this is how the start date filter is done for the existing streams: https://github.com/airbytehq/airbyte/blob/e19c498b00ed4d0fed7e57ecb36996dd2e851344[…]ns/connectors/source-quickbooks/source_quickbooks/manifest.yaml
Copy code
WHERE Metadata.LastUpdatedTime > '{{ stream_slice.start_time }}'
m
okay I will follow it. Thank you very much.
Open in Slack
PreviousNext
Powered by