Is there documentation or recommendations from Airbyte on how to parallelize a single data source? I have a MS SQL source to Snowflake connection with about 2 billion rows which is unable to complete in under 18 hours which as about the max time I’m able to connect to the MS SQL server to perform the backfill. The whole system is running on AWS EC2 at the moment with a 2 core instance, 16GB memory, and 100GB of EBS storage.

if it’s historical data/a one-time sync, just split it in more jobs by tables. Or dump the tables to an S3 bucket and load them from there instead.

So just create multiple connections with the same source and destination and partition which tables sync from each connection? Also, in-terms of the S3 staging, do you know if it is using Snowpipe under the hood and if this improves the overall throughput?

“So just create multiple connections with the same source and destination and partition which tables sync from each connection?” yes, that is what I meant.

Awesome. Thank you for the recommendations here!