We're using pagination on one of our custom connector. However, the API used in this connector has a very strict rate limit (one request per minute), thus the 2nd requests fails. We tried adding an Error Handler to retry the 2nd request after one minute but it'd seem that Airbyte retries from the start, thus we get the error again on the 2nd request. Is this the expected behaviour? Or did we miss something on our setup?

Based on the information provided in the extracts, it seems that Airbyte does have built-in mechanisms to handle rate limits and retries. However, the default behavior is to retry requests that received a 429 and 5XX status code in the response 5 times using a 5-second exponential backoff. This might not be suitable for your case where the API has a strict rate limit of one request per minute. In the [Connector Builder UI](https://docs.airbyte.com/connector-development/connector-builder-ui/error-handling), you can configure an error handler with a "Backoff strategy" and "Response filter". When the conditions of the response filter are met, the connector will proceed with the sync according to the behavior specified. The backoff strategy determines how long the connector should wait before attempting the request again. If you're using the CDK, you can override the

should_retry

and the

backoff_time

methods to customize the retry behavior. Here is an [example](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py#L72) from the Slack API. However, it's important to note that Airbyte will always attempt to make as many requests as possible and only slow down if there are errors. It is not currently possible to specify a rate limit Airbyte should adhere to when making requests. If you've already set up an error handler with a backoff strategy of one minute and the connector is still retrying from the start, it might be a good idea to [create an issue](https://github.com/airbytehq/airbyte/issues/new/choose) to get further assistance from the Airbyte team.

@kapa.ai so it is the expected behavior that Airbyte retries from the 1st page instead of only retrying the page that yielded the error?

Hmm, I don't know enough to give you a confident answer yet. The extracts provided do not contain specific information about Airbyte's behavior in terms of retrying from the first page or the page that yielded the error. However, Airbyte does have a robust error handling mechanism that includes retrying failed requests, with the specifics depending on the response from the partner API and the error handler configuration. For more details, you might find the [Error Handling](https://docs.airbyte.com/connector-development/connector-builder-ui/error-handling) section of the Airbyte documentation useful.