Hi everyone, I have tried to connect the Amazon A...
# replication-troubleshooting
k
Hi everyone, I have tried to connect the Amazon Ads API to BigQuery. There is a large amount of data and the first attempts to sync failed after 24h. The first approach was to limit the sync based on profiles. The sync of 100 profiles also stopped after several hours. Transferring 10 profiles took about 5h (1,200,000 records; 2,2 GB). We currently deployed Airbyte on one VM, so the next step was to scale up the VM to different configurations. Unfortunately, this did not improve the performance. The goal is to sync about 3000 profiles daily. Has anyone had a similar problem before, or a suggestion on how to sync such a large amount of data from the Amazon Ads API most efficiently?
plus1 7
u
Katja would be possible to split the connection into multiple ones and run and parallel to ingest the data? (sorry I have minimal context about Amazon Ads) Saying this because some sources has quota or request limitation. As pointed in Amazon docs: https://advertising.amazon.com/API/docs/en-us/info/developer-notes about some performance consideration to generate the reports.
k
Hi, thanks for your reply! We tried it with 100 and 10 profiles. With 10 profiles around 2.2 GB was transferred, which took between 3-5 hours, depending on the config of the VM. Since more than 3000 profiles have to be transferred, we are looking for a way that requires minimal manual effort.
c
Would it be possible to create more powerful VM?
u
Probably this a limitation from the API side. Using a more powerful server won't increase the number allowed by Amazon. Katja you can use the API to create separate connectors to your 3k profiles.
k
Thanks for the Input!