<@U04S8PLJWMT> i am building a custom conector usi...
# ask-ai
v
@kapa.ai i am building a custom conector using the UI. my source app only returns 25 objects per page, but I need it to return everything by default, the page = 1, if I want to pass to the next one, I need to add a query paramater =2, and so on, until no more data (with the default format) is returned. how can I set this?
k
Use Page Increment pagination in the Connector Builder. From your description: - Fixed page size: 25 objects per page (API default) - First page is
page=1
- Next pages are
page=2
,
page=3
, … - Stop when a page returns fewer than 25 (or no) records Configure the stream’s Pagination section like this: 1. Mode:
Page Increment
[[Page increment](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#page-increment)] 2. Page size:
25
(or omit if the API does not accept a page size parameter) [[Page increment](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#page-increment)] 3. Start from page:
1
(because your API is one-based) [[Advanced page options](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#advanced-page-increment-options)] 4. Inject page number into outgoing HTTP request: - Inject into:
request_parameter
- Field name: whatever your API uses (e.g.
page
) 5. (Optional) Inject page size into outgoing HTTP request if your API supports it: - Inject into:
request_parameter
- Field name: e.g.
page_size
or
per_page
With that, Airbyte will call: -
...?page=1
→ records -
...?page=2
→ records -
...?page=3
→ … - …until a page returns fewer records than the page size or empty, then it stops. [[Page increment](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#page-increment)]
v
@kapa.ai What if I want to set the pagination using a value present in the response body? Every response gives me $.links.next, and this is the link that should be called
k
A message from kapa.ai
Use Cursor Pagination and read the next URL from the response body. In the Connector Builder UI, for your stream: 1. Enable Pagination and choose Pagination method: Cursor Pagination. 2. Where is the next page cursor found? - Select response (since
$.links.next
is in the body). - Property path:
links, next
(Connector Builder expects a comma‑separated path for nested fields, like the Twitter
meta, next_token
example). [[Cursor pagination](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#cursor-pagination)] 3. Inject cursor into request: - Inject into: path - This makes Airbyte replace the entire request URL with the value from
links.next
on subsequent calls, which is exactly the pattern used for APIs that return a full “next page” URL (same as the GitHub example). [[Cursor pagination](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#cursor-pagination)] Airbyte will then: - Call your initial URL (without cursor) - Read
response.links.next
- Call that URL as the next request - Continue until
links.next
is missing/empty, at which point it stops. [[Cursor pagination](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#cursor-pagination)]
v
@kapa.ai if my source application has a limit of 120 requests per minute, where should I set it?
k
A message from kapa.ai
In the Connector Builder UI, you set this in the Global Configuration → HTTP API Budget section. For a limit of 120 requests per minute, configure an HTTP API Budget with a Moving Window Call Rate Policy, for example: - Go to your connector → Global Configuration → HTTP API Budget. [[Global config](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/global-configuration#http-api-budget)] - Add a Moving Window Call Rate Policy with: - `limit`:
120
- `interval`:
PT1M
(ISO-8601 for “1 minute”) [[Rate limiting](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/rate-limit-api-budget); [Rate object](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/Rate)] In YAML (for reference, same concept behind the UI):
Copy code
yaml api_budget:   type: "HTTPAPIBudget"   policies:     - type: "MovingWindowCallRatePolicy"       rates:         - limit: 120           interval: "PT1M"       matchers:         - method: "GET"           url_base: "<https://api.example.com>"
This tells Airbyte to allow at most 120 calls in any 1‑minute window and automatically throttle requests to stay within that budget. [[Rate limiting](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/rate-limit-api-budget); [HTTP API Budget UI](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/global-configuration#http-api-budget)]