Hi everyone,
I have tried to connect the Amazon Ads API to BigQuery. There is a large amount of data and the first attempts to sync failed after 24h.
The first approach was to limit the sync based on profiles. The sync of 100 profiles also stopped after several hours. Transferring 10 profiles took about 5h (1,200,000 records; 2,2 GB).
We currently deployed Airbyte on one VM, so the next step was to scale up the VM to different configurations. Unfortunately, this did not improve the performance.
The goal is to sync about 3000 profiles daily.
Has anyone had a similar problem before, or a suggestion on how to sync such a large amount of data from the Amazon Ads API most efficiently?