I have a connection that is failing because the so...
# replication-ask-ai
d
I have a connection that is failing because the source API says “Result set size is greater than the maximum limit.” How can I tune the page size or what other options are there to get this to pull through. It’s the Paypal Source and I think a lot of transactions came through at once, so changing the date range may not help much.
k
New message text here
d
@kapa.ai where can you make these changes in Airbyte Cloud?
k
New message text here
d
@kapa.ai You said: “If you’re running into these limits, you might need to adjust your sync strategy. For example, you could try syncing smaller date ranges or reducing the number of records you’re trying to pull in a single request or page.” But I don’t think it’s possible to “try syncing smaller date ranges”. Is that correct?
k
A message from kapa.ai
Yes, you're correct. Based on the provided extracts, Airbyte allows you to adjust the "Data Request Interval (Days)" when setting up certain sources like Google Analytics. This interval can range from 1 to 364 days. However, this doesn't necessarily equate to syncing smaller date ranges, but rather affects the granularity of the data fetched in each request. For instance, if you set the "Data Request Interval (Days)" to 1, the system will fetch data for each day separately. This could potentially lead to more accurate results but might also increase the load on the system and the API, and could lead to duplicated data entries for each day. On the other hand, if you set the "Data Request Interval (Days)" to a larger number (like 364), the system will fetch data for that entire period in one go. This could potentially lead to faster syncs but might also result in data sampling, which could cause inaccuracies in the returned results. So, while you can't directly sync smaller date ranges, you can adjust the granularity of the data fetched in each request, which might help in managing API limits or reducing the load on the system. [Source](https://docs.airbyte.com/integrations/sources/google-analytics-data-api)
d
I submitted a help ticket online
m
Hi @Dale Gilliam. I ran into the same issue. For some days, we have more than 10k transactions and the sync gives error when loading those days. I was wondering if you were able to find a solution to that. my Airbyte version is 0.50.1. I tried with different versions of the PayPal connector, but it didn't work. Thank you.
d
@Murat Cetink Sorry for the delay. I think things pushed through… i had one successful run that pulled ~140K transactions. But it continued to fail more times than not. It would fail on 0 transactions and then later succeed with 0 transactions (our transactions get processed all at the same time each day). Unfortunately, my ultimate solution was to use a different tool for pulling the data. I just made that transition on Friday.
m
No worries. I saw that there is an issue on the Airbyte repository. I took a similar approach as yours instead of waiting for the fix which will probably take months. I'm currently writing an Airflow DAG to ingest the data. Thanks for your return anyway 🙂
👍 1