<@U04S8PLJWMT> what does "Days To Sync If History ...
# ask-ai
m
@kapa.ai what does "Days To Sync If History Is Full" mean for source-sftp-bulk? When do i know that the history is full?
k
The "Days To Sync If History Is Full" setting for the SFTP Bulk source connector controls the lookback window Airbyte uses to determine which files to sync when the state history is full. Specifically, Airbyte tracks the files it has already synced in a "history" section of the connection's state message, with a limit of up to 10,000 files. Once this history is full (i.e., it contains 10,000 entries), Airbyte will drop the oldest entries and, for subsequent syncs, will only read files that were last modified between the date of the newest file in history and the number of days specified by "Days To Sync If History Is Full" prior to that date. This mechanism ensures that, even when the state history cannot track every single file (due to the 10,000 file limit), Airbyte will still attempt to capture any changes within a recent window of time, as defined by this setting. You know the history is full when the number of tracked files reaches 10,000, at which point this lookback window is applied automatically to determine which files to sync going forward [Source: S3 State documentation, which is referenced in the SFTP Bulk setup guide](https://docs.airbyte.com/integrations/sources/s3#state). For SFTP Bulk, this setting is available as an optional configuration when setting up your stream in Airbyte Cloud [Setup guide for SFTP Bulk](https://docs.airbyte.com/integrations/sources/sftp-bulk#for-airbyte-cloud).
m
@kapa.ai how to only read files that changed since the last sync?
k
A message from kapa.ai
To only read files that changed since the last sync with the SFTP Bulk source in Airbyte, you should use the incremental loading feature. This feature allows Airbyte to sync files from the SFTP server to your destination based on their creation or last modification time. Airbyte tracks which files have already been synced and, during each sync, only processes files that have been added or modified since the previous sync. When setting up your SFTP Bulk source, you can optionally provide a Start Date. Incremental syncs will only sync files modified or added after this date. For subsequent syncs, Airbyte will use the last sync state to determine which files are new or have changed, ensuring that only those are read and processed in each run [Explore SFTP Bulk features](https://docs.airbyte.com/integrations/sources/sftp-bulk) [Setup guide for SFTP Bulk](https://docs.airbyte.com/integrations/sources/sftp-bulk#for-airbyte-cloud). This incremental approach is efficient and prevents reprocessing files that have not changed.