Hey all! I raised this briefly as a 'fun thing to share' at work. The #1 question I received was:

What scale is this intended for?

For context, we often move ~100GB sets, occasionally up to ~10TB, generally between databases, sometimes through file stores (Azure Storage, SFTP, GCS) but sometimes from our events system too.

just by curiosity, what do you usually use to move these volumes around?

hi James! is this scale on a daily cadence? we’re currently focused on providing a great experience for smaller scales, a few gb per day at the moment, and will set our sights on large scale support in our longer term roadmap. It’s possible to support larger scales today by vertically scaling your Airbyte node, but we don’t currently include that kind of scale as part of our KPIs.

Chris — Currently these are run as daily (or more frequent, but lower volume) Spark jobs. Shrif — Thanks for that, makes perfect sense.