Channels
j
Jackson Clarke
02/07/2022, 9:15 AMHello..I am developing custom connector using python CDK. Having implemented Source ABC, got stuck with incremental sync. Wanted to check how to do incremental sync. Should I implement AbstractSource rather than the Source? Please suggest.
n
Narender Kumar
02/07/2022, 10:09 AMHey would suggest you to go through connections like shopify, google_ads which have incremental implemented also you can look at our doc https://docs.airbyte.com/connector-development/cdk-python/incremental-stream
@Harshith (Airbyte) Can you please clarify the following query:
When I look at AbstractSource class, discover and read functions uses streams function for processing, How do it distinguish that the discover is for getting meta data where as read is to read the data of meta data.
Same confusion is with Salesforce connector.
When you use CDK we handle all of this internally and you can find the linking to these functions in like source.py files
1. https://github.com/airbytehq/airbyte/blob/b447bb5e922b5faf260afe9891dcf038523ad5a3[…]ations/connectors/source-salesforce/source_salesforce/source.py for checking
2. https://github.com/airbytehq/airbyte/blob/b447bb5e922b5faf260afe9891dcf038523ad5a3[…]ations/connectors/source-salesforce/source_salesforce/source.py Read function
3. https://github.com/airbytehq/airbyte/blob/b447bb5e922b5faf260afe9891dcf038523ad5a3[…]ations/connectors/source-salesforce/source_salesforce/source.py Stream schemas
2 Views