Hey, anyone have any tips on how to make the data transfer process faster? I have just recently set up my first connector between Stripe and BigQuery, and even being hosted on the GCP Compute Engine, it seems to take about 10 minutes for 10k rows. I have 500k rows in Stripe, so the whole process will take about 8 hours. I'm not sure if this is just to be expected or if there are ways to speed this up.
For reference, I am using a "e2-standard-4" compute engine, and CPU utilization has only been hovering around 2.5% while the job has been running. Thanks!
Emeka Boris Ama
01/06/2022, 2:10 PM
Hi @Sean McKeever, the bottleneck is probably on Stripe's API side. They have a rate limit that the connector must respect to consume all the source data. Could you please share your sync log so that we can confirm this hypothesis?
@[DEPRECATED] Augustin Lafanechere thanks for the quick response! Here are the logs from the successful job