https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> in UI builder , how does the increm...
# ask-ai
p
@kapa.ai in UI builder , how does the incremental sync work? What should the parameter in the inject be?
k
Incremental sync in Airbyte's UI builder works by pulling only the data that has changed since the previous sync, instead of all the data available in the data source. This is especially useful when there are a large number of records to sync or when the API has tight request limits, making a full sync of all records on a regular schedule too expensive or too slow. To configure incremental syncs for a stream in the connector builder, you need to specify how the records will represent the "last changed" / "updated at" timestamp, the initial time range to fetch records for, and how to request records from a certain time range. In the builder UI, these things are specified as follows: - The "Cursor field" is the property in the record that defines the date and time when the record got changed. It's used to decide which records are synced already and which records are "new". - The "Datetime format" specifies the format the cursor field is using to specify date and time. - "API time filtering capabilities" specifies if the API allows filtering by start and end datetime or whether it's a "feed" of data going from newest to oldest records. - The "Start datetime" is the initial start date of the time range to fetch records for. When doing incremental syncs, the second sync will overwrite this date with the last record that got synced so far. - The "End datetime" is the end date of the time range to fetch records for. In most cases, it's set to the current date and time when the sync is started to sync all changes that happened so far. - The "Inject start/end time into outgoing HTTP request" defines how to request records that got changed in the time range to sync. In most cases, the start and end time is added as a query parameter or body parameter. For example, if the API supports filtering using the request parameters 
created[gte]
and 
created[lte]
, then the component can specify the request parameters as:
Copy code
yaml incremental_sync:     type: DatetimeBasedCursor     <...>     start_time_option:       type: RequestOption       field_name: "created[gte]"       inject_into: "request_parameter"     end_time_option:       type: RequestOption       field_name: "created[lte]"       inject_into: "request_parameter"
In this example, 
inject_into
is set to 
"request_parameter"
, which means the start and end time will be added as a query parameter in the HTTP request. For more details, you can refer to the [Airbyte documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#configuration).
p
Is the cursor field any name or should be from the response ? And the parameter name also? @kapa.ai
k
A message from kapa.ai
The "Cursor field" should be a property in the record from the response that defines the date and time when the record got changed. It's used to decide which records are synced already and which records are "new". This field is typically something like an 
updated_at
timestamp or an 
id
that increments with each new record. For example, if the structure of a stream is 
{ value: 2, metadata: { updated_at: 2020-11-01 } }
, the 
default_cursor_field
might be 
["metadata", "updated_at"]
. This means that the 
updated_at
field in the 
metadata
object is used as the cursor field ([source](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol#cursor)). The parameter name in the "Inject start/end time into outgoing HTTP request" section should correspond to the parameter name that the API expects for filtering records based on time. This will depend on the specific API you are working with. For example, some APIs might expect parameters like 
start_time
and 
end_time
, while others might use different parameter names. You should refer to the API documentation to find out the correct parameter names to use. Here's an example of how you might specify these in the connector builder:
Copy code
yaml incremental_sync:     type: DatetimeBasedCursor     <...>     start_time_option:       type: RequestOption       field_name: "start_time"  # replace with the correct parameter name for the API     inject_into: "request_parameter"     end_time_option:       type: RequestOption       field_name: "end_time"  # replace with the correct parameter name for the API     inject_into: "request_parameter"
In this example, 
start_time
and 
end_time
are the parameter names that the API expects for filtering records based on time. These should be replaced with the correct parameter names for the API you are working with ([source](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#configuration)).
Open in Slack
PreviousNext
Powered by