I have an S3 data source in Airbyte Cloud. Every day a full overwrite data set is delivered to this bucket, under a directory with the date and time. I can successfully consume the first file, but is there a way to either only retrieve the most recently modified file, or to use something like a current date in the globs pattern?

Thinking completed I'm sorry, but the knowledge sources do not explicitly describe a built-in feature to dynamically inject "current date" variables into the glob pattern or to configure the connector to only retrieve the single most recently modified file while ignoring earlier ones. However, based on the provided documentation, here is how the S3 source works and how you can approach this: 1. Glob Patterns Do Not Support Dynamic Dates or Regex Variables: Airbyte’s S3 connector path matching uses standard glob syntax via

wcmatch.glob

[Globs]. It does not support regex-based date evaluation or dynamic expressions like today's date [Forum Post: Issue with regex in S3 connector setup]. 2. Using Incremental Sync for New Files: Instead of filtering by today's date in the glob pattern, you can use a broad glob (for example,

**/*.csv

or

*/*/*.csv

) and configure your connection sync mode to Incremental Sync [Supported sync modes]. ◦ When configured for incremental sync, Airbyte processes files from oldest to newest based on their last-modified timestamp [State]. ◦ Once a file is processed, its record is tracked in the connection's state history [State]. ◦ In subsequent sync runs, Airbyte will skip the files it has already synced and only read the newly added or updated files that were modified after the previous sync cursor [State; Airbytehq Airbyte > Blob > Master > Airbyte-integrations > Connectors > Source-s3 > Source_s3 > stream.py>]. 1. Handling Destination Overwrites: If your daily incoming files represent a complete snapshot and you want only the latest day's data stored in your destination, note that setting the connection to sync incrementally will still append the new snapshot records into your destination tables. The knowledge sources do not provide a mechanism within the S3 source itself to automatically discard earlier files if you run a Full Refresh Overwrite sync. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Why is it not stated in the regular documentation that an incremental sync will only read newly added or updated files? Did I miss something?

Thinking completed You may have missed it, but this behavior is covered across both the connector-specific documentation and Airbyte's core concepts: 1. Connector-Specific State Tracking: In the State section of the S3 connector documentation, it explains that incremental syncs work by tracking synced files in the connection state: "To perform incremental syncs, Airbyte syncs files from oldest to newest. Each file that's synced (up to 10,000 files) will be added as an entry in a "history" section of the connection's state message." [State] Under the hood, the S3 connector tracks each file's _ab_source_file_last_modified date and checks incoming files against this state history to avoid re-reading files that have already been processed [Forum Post: S3 / CSV Incremental strategy; Airbytehq > Airbyte > Blob > Master > Airbyte-integrations > Connectors > Source-s3 > Source_s3 > stream.py]. 2. General Incremental Sync Concept: In Airbyte's core Sync Modes documentation, the definition of an Incremental read is: "Incremental: Read records added to the source since the last sync job." [Sync Modes] Additionally, the general platform docs clarify that incremental syncs pull only data that has changed since the previous sync using a cursor value (such as a timestamp) to determine which records or files are new [Incremental Sync]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.