Hi All, we are looking to sync a 1B+ row table from MySQL using CDC Incremental Append. We have successfully sync'd 100M row datasets from MySQL but the loading time is a 1-2 days as far as I remember. This won't allow us enough time to catch up with the binlog. I don't need the entire table, and would be happy with the last year or two. I'm wondering if anyone has strategies for syncing such large tables? I can see that the M*S*SQL connector allows CDC to be configured to 'New Changes Only' - skipping the initial snapshot. Is there a way to achieve this with the MySQL connector? I'm happy to capture new changes only and then I can complete a full load of a view which restricts the data to the last year or two and combine the two in Snowflake. Would love to hear any experiences people have had in this space.

Hi Adam, the MySQL connector does not currently have that functionality but you can certainly make a feature request. Have you seen the doc on scaling? That might be helpful here: https://docs.airbyte.com/operator-guides/scaling-airbyte

Nataly, is there any way to scale the number of source workers on a single table read? Ie, read in parallel? If he could hash a column in his 1B+ row table into 100 chunks, he could have 100 workers each reading 10k rows at a time.

I don't believe there is ability to read in parallel, have you gotten a chance to look through the scaling doc?

I did yes. For some of the larger, more oft used data sources, it might be good to at least table the idea of adding parallel reads. Postgres, Oracle, MSSQL, maybe even Redshift, Snowflake, BigQuery. Incremental append is always good, CDC is better, but that first load can take forever without parallelization.

For any feature requests please open an issue in GitHub!

Hi @Nataly Merezhuk (Airbyte), thank you for your reply. I have seen the scaling doc, and I think we are spec'd well enough - we are running on a c6a.2xlarge on AWS, which is 16gb ram and 8 cores. We only tend to run 1-2 jobs at a time and from what I can see we never exhaust the ram. I think our bottleneck would be either in the source database, or the network bandwidth? I will submit a feature request for the 'New Changes Only' feature from MSSQL. @Jordan Fox I think parallel reads would be really helpful! Even regular table loads can take a couple hours and speeding these up would be very useful. I wonder if anyone has raised the idea before?

I'm sure they have Adam, but I'll search through the issues list some time this month over holidays to confirm. I'm currently writing my own Azure Blob Destination connector too since it doesn't parallel write and on incremental appends it'll write a new blob regardless of whether the stream has any data (5 min sync schedule creates 0 byte blobs every 5 mins).

Thanks @Jordan Fox that's helpful. I had a search for the 'New Changes Only' / CDC config, but couldn't find one, so I have created a request : https://github.com/airbytehq/airbyte/issues/20112 0 byte blobs is not ideal! Hope you make some good progress 🙂 I had a look for parallel read requests and came across these: https://github.com/airbytehq/airbyte/issues/7750 https://github.com/airbytehq/airbyte/issues/7749 https://github.com/airbytehq/airbyte/issues/4081

Thanks for opening that feature request, Adam! I've triaged it!

Thanks @Nataly Merezhuk (Airbyte) 🙂