Hello team,
I am trying to move a very big single table from PostgreSQL to Snowflake (1,800 GB). Already managed to move several other tables between the same source and destination.
Airbyte deployment is based on K8S, all resources (source, destination and Airbyte) are in AWS us-east-1
I am currently trying for several days without any luck, two things come to mind:
1. It seems that none of the resources are near their memory/CPU limits yet Airbyte seems to be able to move only around 30GB/hour - it seems that the fact it reads only 10,000 records at a time should be configured to a higher rate. Any way to do that?
2. It seems that a random glitch in either source/destination will cause the entire attempt to fail - the odds of that highly increase on an approximate 100-hour transfer and yet it can never finish (usually fails after reading somewhere between 20GB and 200GB, failure is sometimes due to source and sometimes due to destination) - any way to let it continue from the same point and not restart from scratch? (Incremental sync - deduped + history - single table)