Hi all, We’ve been struggling with a pretty simple issue that doesn’t seem to have a clear solution in Airbyte, Initial loading of big tables from a RDS (PG or mySQL) to a DWH (Snowflake). Airbyte just can’t overcome the initial load of a big table and fails over and over again. I thought about getting the initial load by exporting it into a file and loading it into Snowflake but: 1. I’m not sure I can create the target table for Airbyte. 2. Target table usually contains internal AB id’s I can’t generate. 3. I’m pretty sure Airbyte won’t recognize the increment and we’ll try to load the entire data set again. How did you overcome that issue? Any creative workaround would be gladly accepted. Thanks!

Hi Dudu! I don't have a creative workaround in mind for you yet, but could you please give me more specifics about the tables? Do you know the average row sizes for these tables? I'm looking at this and wondering if it's under the limit stated: https://airbyte7538.zendesk.com/agent/tickets/3363

Hi @Nataly Merezhuk (Airbyte) The link you’ve provided requires login. Can you get a quote of the limit stated? Is that mentioned anywhere in the formal documentation? Thanks.

Apologies, here is the correct link: https://airbytehq.github.io/operator-guides/scaling-airbyte/#memory I'd say the best platform for your question would be a GitHub feature request! That way you'd get more input from engineering teams on workarounds and potential plans to address this in our roadmap.