Hello everyone,
We are running Airbyte as a docker-compose application on a VM (GCE) in order to synchronise a large amount of data from Salesforce to BigQuery.
The first approach was to use the Salesforce source and BigQuery destination, however the synchronisation process took 8 days (for a single table) before crashing due to a 400 Bad Request to the Salesforce API.
Our issues:
• Is there a way to ensure (or check) Airbyte will start the next sync where the last crashed ?
• When using other destinations than BigQuery (ie: GCS with parquet files), it seems that Airbyte workers try to load really big chunks of data (currently > 10GB, and it's still increasing) before loading them to the destination (GCS)...Is there a way to customize this behavior to a more stream-like or mini-batch behavior ?