Hey all 👋 I’m looking into how to get data from S3 properly, using the S3 source connector 👀 In S3, we have a full snapshot of a third party system dumped every day, and named after the day for which the dump is (so basically

<s3://my-bucket/some-path/dataset1-2023-01-17.csv>

for today, etc). As you probably know, I need to provide a regex to define which file(s) to pull. What I’d like to do, using the S3 connector, is to only read the file for the current day, rather than all the files present there for

dataset1

(i.e.

dataset1-*.csv

does not work and something like

dataset1-<some_date>.csv

would work). I thought about going incremental but since each day is a full snapshot, I think this is somewhat overkill and the “state” does not make sense in that situation. Is there a way to achieve this with Airbyte and the S3 connector ? Let me know if there’s any additional detail I can provide. Thanks!

Hey Antoine, I think this should be achievable. Let me tinker with some patterns and circle back to you.

Sounds good, thanks for looking into this @Ryan (Airbyte) 🙇

Hey, so I might have 🦶 😮 The new low-code framework allows for some macros, which I thought could be used here. While that is true, the S3 Source hasn't been migrated to that framework, so no luck there. As for alternatives until macros are supported. The best (and only) idea I have at the moment is to leverage the API to make calls to

/v1/connections/update

endpoint with an update to the pattern option. Thats a pretty rough solution, but I suspect if integrated into an orchestrator it could be made a little more resilient. In the meantime I've +1'd a feature request for macros in the S3 source, agnostic of code framework.

Thread sent successfully to Harvestr

Hey @Ryan (Airbyte), thanks for the feedback 🙇 I’ll check in with the team what the options are on our side, and keep a close eye on that feature request. Thanks!

Feedback was linked to File-based CDK: Research Data

Linked feedback:

New Feedback