Hello @kapa.ai I want to download a big table from Postgres, but I want to sync from a certain date based on a field. Is there a way to do this on airbyte

In Airbyte, you can achieve this by using the Incremental Sync feature. Incremental syncs pull only the data that has changed since the previous sync, which is especially useful when dealing with large datasets. This is usually implemented using a cursor value, like a timestamp, that delineates which data was pulled and which data is new. To use incremental syncs, the API endpoint needs to fulfill the following requirements: - Records contain a top-level date/time field that defines when this record was last updated (the "cursor field"). - It's possible to filter/request records by the cursor field. In your case, you can set the cursor field to the date field you want to start syncing from. Records whose date value is less than or equal than that cursor value have been synced already, and the next sync should only export records whose date value is greater than the cursor value. Here is an example of how you can set up an incremental sync: 1. Define the name which will identify this connection. 2. Select the incremental append replication mode for the table. 3. Select your date field as the cursor for the table. After you set up the connection, the initial sync will start. Once it completes, subsequent syncs will only pull the data that has changed since the previous sync. Please note that not all sources and destinations support checkpointing, a feature that allows Airbyte to restart from a known point in case of a network error or a server going down for maintenance during a sync operation. This feature is currently being worked on by Airbyte. For more information, you can check out the [Incremental Sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync) and the [Postgres source documentation](https://docs.airbyte.com/integrations/sources/postgres#postgres).

A message from kapa.ai

In Airbyte, you can achieve this by using the Incremental Sync feature. Incremental syncs pull only the data that has changed since the previous sync, which is especially useful when dealing with large datasets. This is usually implemented using a cursor value, like a timestamp, that delineates which data was pulled and which data is new. To use incremental syncs, the API endpoint needs to fulfill the following requirements: - Records contain a top-level date/time field that defines when this record was last updated (the "cursor field"). - It's possible to filter/request records by the cursor field. In your case, you can set the cursor field to the date field you want to start syncing from. Records whose date value is less than or equal than that cursor value have been synced already, and the next sync should only export records whose date value is greater than the cursor value. Here is an example of how you can set up an incremental sync: 1. Define the name which will identify this connection. 2. Select the incremental append replication mode for the table. 3. Select your date field as the cursor for the table. After you set up the connection, the initial sync will start. Once it completes, subsequent syncs will only pull the data that has changed since the previous sync. Please note that not all sources and destinations support checkpointing, a feature that allows Airbyte to restart from a known point in case of a network error or a server going down for maintenance during a sync operation. This feature is currently being worked on by Airbyte. For more information, you can check out the [Incremental Sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync) and the [Postgres source documentation](https://docs.airbyte.com/integrations/sources/postgres#postgres).