This message was deleted.

I don't know about airflow specifically, but two approaches that I have heard recommended: • use a lambda to trigger whenever a new file is created in the bucket, place that filename on a queue list somewhere that an ingestion job can periodically pick up for batch ingestion. • smartcode your file names with a date/time prefix (e.g. YYYYMMDD-HH) and set up a job to run at that interval and process all files with the interval prefix.