https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> I'm trying to use the Builder UI/YA...
# replication-ask-ai
y
@kapa.ai I'm trying to use the Builder UI/YAML to create a connector to extract data from an API. I'm facing issues with the Paginator feature: it doesn't append the page number query param (in this case, 
pageNo
) to the request. Is this a known bug or am I doing something wrong? More context: The paginator section of the YAML file:
Copy code
...
paginator:
        type: DefaultPaginator
        page_size_option:
          type: RequestOption
          field_name: pageSize
          inject_into: request_parameter
        page_token_option:
          type: RequestOption
          field_name: pageNo
          inject_into: request_parameter
        pagination_strategy:
          type: PageIncrement
          page_size: 100
          start_from_page: 1
...
I get back a 422, saying that the 
pageNo
param is missing:
Copy code
Request to <https://example.com/api/v2/datafeed?data=endpoint&pageSize=100> failed with status code 422 and error message Page number is required and needs to be greater than 0
and in the request window in the Test UI, you see that 
pageNo
was not sent:
Copy code
{
  "url": "<https://example.com/api/v2/datafeed>",
  "parameters": {
    "data": [
      "endpoint"
    ],
    "pageSize": [
      "100"
    ]
  }
here are the logs (where you can also see that 
pageNo
was not sent):
Copy code
Request to <https://example.com/api/v2/datafeed?data=endpoint&pageSize=100> failed with status code 422 and error message Page number is required and needs to be greater than 0. - Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 249, in _read_stream
    yield from AirbyteEntrypoint(source).read(source.spec(self.logger), config, configured_catalog, {})
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 156, in read
    yield from self.source.read(self.logger, config, catalog, state)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 138, in read
    yield from super().read(logger, config, catalog, state)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 139, in read
    raise e
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 119, in read
    yield from self._read_stream(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 194, in _read_stream
    for record in record_iterator:
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 304, in _read_full_refresh
    for record_data_or_message in record_data_or_messages:
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 104, in read_records
    yield from self.retriever.read_records(stream_slice)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 304, in read_records
    for stream_data in self._read_pages(self._parse_records, self.state, stream_slice):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 282, in _read_pages
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 395, in _fetch_next_page
    return self.requester.send_request(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 399, in send_request
    return self._validate_response(response)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 496, in _validate_response
    raise ReadException(error_message)
airbyte_cdk.sources.declarative.exceptions.ReadException: Request to <https://example.com/api/v2/datafeed?data=endpoint&pageSize=100> failed with status code 422 and error message Page number is required and needs to be greater than 0.
k
New message text here
y
@kapa.ai which available variable is the one that points to page number?
k
A message from kapa.ai
The available variable that points to the page number in Airbyte's connector builder is 
next_page_token
. This variable is used in the pagination strategy to keep track of the next page to be fetched. In your case, you can use the 
next_page_token
variable to configure a query parameter with key 
pageNo
and value 
{{ next_page_token['next_page_token'] or '0' }}
. This will inject the page number from the pagination strategy into the 
pageNo
parameter. Here's an example of how you can use it:
Copy code
yaml parameters:   - name: pageNo     value: "{{ next_page_token['next_page_token'] or '0' }}"
This will ensure that the 
pageNo
parameter is included in the request with the correct page number. For more details, you can refer to the [Pagination documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination/).
Open in Slack
PreviousNext
Powered by