@kapa.ai using cursor pagination to pull from the datadog api, I need to inject the cursor value into the next request, but it's nested inside another element. I've tried specifying the element (

page['cursor']

) and I've tried jinja expressions, but it doesn't work. I also tried putting the value into the json request payload like this:

{
  "page": {
    "limit": 25,
    "cursor": "{% set cursor_value = response['meta']['page']['after'] if response is defined else '' %}{{ cursor_value }}"
  },
  "sort": "-timestamp",
  "filter": {
    "from": 1725163200000,
    "query": "@service:markets-mobile  @application.id:2c03180c-6541-4f87-b1c3-8897485ad074  @session.type:user"
  }
}

withi this, however, the first request fails because the

response

variable isn't defined. Is there any way to access that nested element in the GUI without extending the paginator class?

Yes, you can access the nested element in the GUI without extending the paginator class. You can use the

CursorPagination

strategy in Airbyte's Connector Builder. This strategy allows you to define how the "Next page cursor" is obtained for each request. In most cases, the next page cursor is either part of the response body or part of the HTTP headers. You can configure how the cursor value is injected into the API Requests. For cursor pagination, if

path

is selected as the

Inject into

option, then the entire request URL for the subsequent request will be replaced by the cursor value. This can be useful for APIs that return a full URL that should be requested for the next page of results. Here is an example of how you can define the cursor value in the

CursorPagination

strategy:

yaml paginator:     type: "DefaultPaginator"     pagination_strategy:       type: "CursorPagination"       cursor_value: "{{ response['meta']['page']['after'] }}"   page_token_option:       type: "RequestPath"       field_name: "cursor"       inject_into: "request_parameter"

In this example, the

cursor_value

is set to

{{ response['meta']['page']['after'] }}

, which is the path to the cursor in the response body. The

field_name

is set to

"cursor"

, which is the name of the request parameter that should be set to the cursor value. The

inject_into

option is set to

"request_parameter"

, which means that the cursor value will be set as a request parameter in the next request. Please note that the actual path to the cursor in the response body and the name of the request parameter might be different in your case. You should adjust these values according to the API documentation of the data source you are working with. For more information, you can refer to the [Airbyte documentation on Cursor Pagination](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#cursor-pagination).

for this API, the cursor value needs to be injected into the body json. ie the yaml would look like this:

paginator:
          type: DefaultPaginator
          page_token_option:
            type: RequestOption
            inject_into: body_json
            field_name: page['cursor']
          pagination_strategy:
            type: CursorPagination
            cursor_value: "{{ response.get(\"meta\", {}).get(\"page\", {}).get(\"after\", {}) }}"
            stop_condition: >-
              {{ not response.get("meta", {}).get("page", {}).get("after", {})
              }}

however, the

field_name

is expecting a top-level element and interpolates the string to look for a field literally called "page['cursor']". Is there a way to put a nested element in the

field_name

?

Sorry @Kolin Ohi, I'm not sure what you mean. Please try again. 👋

@kapa.ai for this API, the cursor value needs to be injected into the body json. ie the yaml would look like this:

paginator:
          type: DefaultPaginator
          page_token_option:
            type: RequestOption
            inject_into: body_json
            field_name: page['cursor']
          pagination_strategy:
            type: CursorPagination
            cursor_value: "{{ response.get(\"meta\", {}).get(\"page\", {}).get(\"after\", {}) }}"
            stop_condition: >-
              {{ not response.get("meta", {}).get("page", {}).get("after", {})
              }}

however, the

field_name

is expecting a top-level element and interpolates the string to look for a field literally called "page['cursor']". Is there a way to put a nested element in the

field_name

?

A message from kapa.ai

Based on the information provided in the knowledge sources, it seems that Airbyte currently does not support nested cursor fields for pagination in the Connector Builder UI. This limitation is mentioned in several discussions and issues ([source 1](https://github.com/airbytehq/airbyte/discussions/44913), [source 2](https://github.com/airbytehq/airbyte/issues/36253), [source 3](https://discuss.airbyte.io/t/788), [source 4](https://github.com/airbytehq/airbyte/issues/15737)). In the Connector Builder, the Key Name field only asks for a key and not the path to the key. Therefore, you cannot modify a nested field for pagination directly in the UI. However, there are some workarounds suggested in the discussions. One of them is to move the cursor field to the root of the record. This can be done using an [AddField transformation](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#adding-fields) ([source 4](https://github.com/airbytehq/airbyte/issues/15737)). Another workaround is to manually modify the request body to include the pagination token. This can be done by disabling the "Inject" options under Pagination and manually adding the keys and values in the

Request Body

field. You can use Jinja templating to include the

next_page_token

in the request body ([source 1](https://github.com/airbytehq/airbyte/discussions/44913)). Please note that these workarounds might not be applicable in all cases and might require additional modifications based on the specific requirements of the API you are working with.

@kapa.ai is there a way to apply an add field only after the first request? I'm trying to add this value

"{% set cursor_value = response['meta']['page']['after'] if response is defined else '' %}{{ cursor_value }}"

but the

response

variable isn't defined in the first request

A message from kapa.ai

Based on the information provided in the knowledge sources, it seems that Airbyte currently does not support conditional application of transformations based on the request number. The transformations are applied to all records uniformly. However, you can use a conditional statement in your Jinja template to handle the case where the

response

variable is not defined. In your case, you are already using a conditional statement to set the

cursor_value

to an empty string if the

response

variable is not defined. This should prevent any errors from occurring when the

response

variable is not defined in the first request. Here is the Jinja template you provided:

"{% set cursor_value = response['meta']['page']['after'] if response is defined else '' %}{{ cursor_value }}"

This template sets

cursor_value

to

response['meta']['page']['after']

if the

response

variable is defined, and to an empty string otherwise. Then it outputs the value of

cursor_value

. If you are still encountering issues with this template, it might be due to other factors not covered in the knowledge sources. In that case, I would recommend reaching out to the Airbyte community or support for further assistance.