https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
n

Nathan Atkins

07/23/2021, 2:23 PM
Hello, One of my use cases is the following: we're aggregating data for different customers, and a lot of them deliver data daily by files. For now we'll use File sources and update the link manually every time we know we have new data, but I was wondering if that's maybe something that could be improved once Webhooks are implemented, namely: posting to a hook that a new file is available (with data about how to deal with it) on file upload? Or is it a different kind of webhook? Thanks!
i

Isaline L.

07/23/2021, 3:24 PM
Hey @Isaline L. 👋 I've been working on a new files source which handles multiple files and incrementally loading only new files over time. The MVP release should be quite soon now (supporting CSV format on S3, with plans to add other formats and providers soon after). Does this sound like something that would solve your use-case or are there extra concerns. Would be keen to have your input 🙂
u

user

07/23/2021, 3:42 PM
Hey @George Claireaux (Airbyte)! You mean that people could just add their files periodically and the source would know which ones are already be loaded? Because if so, that sounds pretty good too!
u

user

07/23/2021, 3:43 PM
Yup 😄
u

user

07/23/2021, 3:44 PM
🥳
u

user

07/23/2021, 3:44 PM
Is there a way I could help maybe? Like for introducing more file formats?
u

user

07/23/2021, 3:45 PM
There's quite a few similar issues around this but if you want follow progress, this is the main one I'm using: https://github.com/airbytehq/airbyte/issues/2622
i

Isaline L.

07/23/2021, 3:46 PM
Perfect, thanks. 👍
u

user

07/23/2021, 3:47 PM
Absolutely! Once the main connector is released, it should be comparatively easy to add extra support for new file formats / storage providers based on the abstract framework it'll ship with 🙂
u

user

07/23/2021, 3:48 PM
(and I'm happy to support your efforts if you decide to help out!)
u

user

07/23/2021, 4:14 PM
Great, looking forward to it! 🙂
u

user

07/30/2021, 3:34 PM
Hey again @Isaline L.! Reaching out since this PR is now merged 😄 We'd really welcome contributions on extending file format support! I've created a bunch of issues for them (#5102 - #5109)!
u

user

07/30/2021, 3:35 PM
Let me know if you're interested and I'll be more than happy to help out airbyte rocket
u

user

07/30/2021, 4:42 PM
Oh amazing!! I'm taking some time off next week but I'll try to join the effort as soon as I have a bit of mental bandwidth. Good job in any case! 🥳
u

user

07/31/2021, 4:35 PM
No pressure and no rush! Have a nice time off octavia loves
2 Views