Hi all! I have a POC running to replicate some very large databases (30T+ rows). I gave it a good-sized EC2 host, but I see that it is not using many resources and the max batch size is 10k rows. Should I be upping that for performance?

30 trillion rows?

I'm new here, but we have 15B payment records (recent samples show ~13M/day) and I know our platform includes user-to-user chat, which I have to figure is at least 100:1 ratio to payments... so, something like that.

Hi Edward, unfortunately Airbyte doesn’t handle your level of scale extremely well yet (most of our users range at the ten of thousands to single million rows) so I might not be able to offer a great solution but I’ll do my best

@Davin Chia (Airbyte) do you still believe this to be the case?

We’ve made some improvement here. It would depend on the source and destination. Most of our DB source and destinations should hold up fine but will probably require beefy resources

I’m currently trying to sync MySQL (Aurora) to Snowflake. There are a handful of tables that 10M+ but the rest are relatively small. As you can imagine these are fact tables that grow by some decent amount each day, but nothing too crazy in a steady state. What I’ve been having trouble with is performing the initial historical sync. Using EC2 t3.large (the recommended), I’m hitting like a few cases I’ve seen a Java OutOfMemory on heap size. I changed tack and decide to replicate tables individually however long that would take but then discovered that upon returning to the connector on the Airbyte UI I had to refresh the schema of my source and go through a necessary reset of the source. So it appears it’s not possible to incrementally add tables from a source right now. There was a Github issue that seemed to confirm this. Now the only option that remains it seems is to beef up the EC2 instance, but it’s not clear what rule of thumb or starting point I should adopt with respect to the size of the data in my source. CPU utilization is 100 for the last couple of hours and I’ve decided to just stop the sync. I have some experience with other data loaders and some custom pipelines I’ve created that leverage Snowflake’s COPY INTO to load data from a stage (S3, GCP Cloud Storage etc.) and these seem to be super quick and handle vast quantities of data. It seems Airbyte employs this too and so the “slow” part of this isn’t loading data into Snowflake but getting it out of source.

I would definitely give beefing up the initial instance a shot. Our containers now get 1/4 of the total memory of the instance, so they don’t aren’t using the full instance memory. Better handling this is in the works

OK interesting, I’ll go down that route. Our tables aren’t particularly wide, I mean at most 20 columns on any large table, all very normalised. As for the inc sync, I suppose I hadn’t considered that as an option, but no reason not to. Still thinking with the Fivetran/Stitch mentality here. On speed, yes we are utilising an S3 bucket provisioned for Airbyte for that purpose and the Snowflake query logs do show use of it to load data, although when I go to check S3 there bucket is empty, I suspect that’s because the bucket is used for a relatively short period of time with the loaded data being purged once the transfer to Snowflake is complete? In terms of duration, given what we’ve discussed do you believe such loads should be on the order of hours or days?

Yes. The bucket is purged after used. I’ve seen syncs of this scale take 10 - 15 hours so that’s my hand wavy estimate for big syncs.