https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
n

Nick Miller

12/22/2021, 8:19 PM
Hello, all 👋 ! Loving Airbyte so far. I’m currently prototyping some of our implementations. Specifically planning to use the S3 destination & Incremental Sync to hold the cloned data pending some custom additional processing+analytics. The data size is a concern (ideally I’d be able to cap at X MB or Y records per file) which is currently not supported but the docs do call it out
Currently, each data sync will only create one file per stream. In the future, the output file can be partitioned by size. Each partition is identifiable by the partition ID, which is always 0 for now.
It would be amazing if anyone could point me to updates / roadmaps / possibility of contributing / ability to advocate for this feature. We’ll be using
Incremental - Append Sync
, So any concerns could be mitigated by just skipping/manually doing the initial sync, and then manually setting the cursor field to start at current time, I’m curious if that’s possible/appropriate at all given my file size concern?
4 Views