Hello everyone, We are running Airbyte as a docke...
# replication-troubleshooting
j
Hello everyone, We are running Airbyte as a docker-compose application on a VM (GCE) in order to synchronise a large amount of data from Salesforce to BigQuery. The first approach was to use the Salesforce source and BigQuery destination, however the synchronisation process took 8 days (for a single table) before crashing due to a 400 Bad Request to the Salesforce API. Our issues: • Is there a way to ensure (or check) Airbyte will start the next sync where the last crashed ? • When using other destinations than BigQuery (ie: GCS with parquet files), it seems that Airbyte workers try to load really big chunks of data (currently > 10GB, and it's still increasing) before loading them to the destination (GCS)...Is there a way to customize this behavior to a more stream-like or mini-batch behavior ?
I just found out about the 10k records reading in the source, which explain why the Salesforce source connector keep pulling data which in turn weight a few tenth a GB. source: • https://docs.airbyte.com/operator-guides/scaling-airbyte#memoryhttps://github.com/airbytehq/airbyte/issues/3439 I yet have to found out if this 10K limit can be customized.