Hello, One of my use cases is the following: we're aggregating data for different customers, and a lot of them deliver data daily by files. For now we'll use File sources and update the link manually every time we know we have new data, but I was wondering if that's maybe something that could be improved once Webhooks are implemented, namely: posting to a hook that a new file is available (with data about how to deal with it) on file upload? Or is it a different kind of webhook? Thanks!

Hey @Isaline L. 👋 I've been working on a new files source which handles multiple files and incrementally loading only new files over time. The MVP release should be quite soon now (supporting CSV format on S3, with plans to add other formats and providers soon after). Does this sound like something that would solve your use-case or are there extra concerns. Would be keen to have your input 🙂

Hey @George Claireaux (Airbyte)! You mean that people could just add their files periodically and the source would know which ones are already be loaded? Because if so, that sounds pretty good too!

Yup 😄

🥳

Is there a way I could help maybe? Like for introducing more file formats?

There's quite a few similar issues around this but if you want follow progress, this is the main one I'm using: https://github.com/airbytehq/airbyte/issues/2622

Perfect, thanks. 👍

Absolutely! Once the main connector is released, it should be comparatively easy to add extra support for new file formats / storage providers based on the abstract framework it'll ship with 🙂

(and I'm happy to support your efforts if you decide to help out!)

Great, looking forward to it! 🙂

Hey again @Isaline L.! Reaching out since this PR is now merged 😄 We'd really welcome contributions on extending file format support! I've created a bunch of issues for them (#5102 - #5109)!

Let me know if you're interested and I'll be more than happy to help out airbyte rocket

Oh amazing!! I'm taking some time off next week but I'll try to join the effort as soon as I have a bit of mental bandwidth. Good job in any case! 🥳

No pressure and no rush! Have a nice time off octavia loves