<@U04S8PLJWMT> If the page contains less than 5 re...
# ask-ai
m
@kapa.ai If the page contains less than 5 records, then the paginator knows there are no more pages to fetch. If the API returns more records than requested, all records will be processed. The paginator stops by default if the number of records is less than the max page size - but it stops wrongly in some cases. For example, I have an API that returns max 200 records, it does return 200 for page 1 and 2, then returns 199 (for whatever reason) which stops airbyte, even though there is still a page 3 , 4 , 5 etc after. Is there a way around this?
k
You're correct: by default, Airbyte's paginator for the Page Increment strategy stops fetching more pages if a page contains fewer records than the configured page size. This is intended to signal the end of the dataset, but as you've observed, it can cause premature stopping if the API occasionally returns a page with fewer records even though more pages exist. The knowledge sources do not describe a built-in way to override this default stop condition for Page Increment pagination in the config-based YAML or Connector Builder UI. The default logic is: "If the page contains less than N records, then the paginator knows there are no more pages to fetch" ([source](https://docs.airbyte.com/platform/next/connector-development/config-based/understanding-the-yaml-file/pagination#pagination-strategies)). Workarounds: - If your API provides a more reliable indicator of more pages (such as a "has_more" field, a next page token, or a total page count), you may need to use a different pagination strategy (like Cursor Pagination) or implement a custom stop condition. - For APIs with non-standard or complex pagination logic, Airbyte supports a "Custom Pagination Strategy" where you can write your own Python logic to determine when to stop paginating. This allows you to implement custom rules, such as continuing until a certain page number or until a specific field in the response indicates the end ([Custom Pagination Strategy](https://docs.airbyte.com/platform/next/connector-development/connector-builder-ui/pagination#custom-pagination-strategy)). Summary: With the default Page Increment strategy, there is no configuration option to change the stop condition from "less than page size" to something else. To handle your scenario, you would need to implement a custom pagination strategy in Python, where you can define exactly when to stop fetching pages. If you need help with a custom pagination class, let me know!
m
There is no other way to know from the API. can i maybe use offsetIncrement to avoid pageincrement limitation, and just increment page_number? My api takes a basic page_number request_parameter
@kapa.ai There is no other way to know from the API. can i maybe use offsetIncrement to avoid pageincrement limitation, and just increment page_number? My api takes a basic page_number request_parameter
k
A message from kapa.ai
Based on the knowledge sources, if your API only supports a page number (e.g.,
page
or
page_number
) and not an offset, you cannot use the Offset Increment strategy to increment the page number—Offset Increment is specifically for APIs that use an
offset
parameter, not a page number. The Offset Increment strategy injects an offset value (e.g.,
offset=0
,
offset=5
, etc.), not a page number, into the request ([source](https://docs.airbyte.com/platform/next/connector-development/config-based/understanding-the-yaml-file/pagination#pagination-strategies)). The Page Increment strategy is designed for APIs that use a page number, but as you noted, it stops when a page returns fewer records than the page size, which is a limitation in your case ([source](https://docs.airbyte.com/platform/next/connector-development/config-based/understanding-the-yaml-file/pagination#pagination-strategies)). In summary: - You cannot use Offset Increment to increment a page number parameter. - If your API only supports page numbers, you are limited by the Page Increment strategy's default stop condition. - There is no configuration in the knowledge sources to change this stop condition. If you cannot change the API or use a different pagination indicator, the only alternative (not covered in the provided knowledge sources) would be to implement a custom pagination strategy, but this is not described in detail in the sources above. Conclusion: You cannot use Offset Increment to work around the Page Increment limitation for a page number parameter. The knowledge sources do not provide a solution for your scenario.