Ben Rohald
10/29/2024, 12:05 AM"Limit": "{{ next_page_token['next_page_token'] }}"
into my request. Even when I have pagination enabled, the variable is empty when sending the requestuser
10/29/2024, 12:06 AMnext_page_token
variable in Airbyte's connector builder. This is a known issue that has been reported and discussed. Let me provide some insights and potential solutions: 1. The issue you're experiencing is similar to what was reported in [this GitHub issue](https://github.com/airbytehq/airbyte/issues/40697). The next_page_token
variable may not be interpolated correctly in certain scenarios, especially when used in a JSON request body. 2. One potential workaround is to use form-encoded data instead of JSON for the request body. As suggested in a [GitHub discussion](https://github.com/airbytehq/airbyte/discussions/44913), you can: - Disable the "Inject" options under Pagination - Change the Request Body
field to Form encoded (key-value pairs)
- Add your keys and values, using Jinja templating for dynamic values 3. For your specific case, you might try something like this: Key: "Limit" Value: "{{ next_page_token.next_page_token|default('0') }}"
4. If you need to use JSON, you can try injecting the next_page_token
into a query parameter instead of the request body. The [documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#custom-parameter-injection) suggests: "For example the Prestashop API requires to set offset and limit separated by a comma into a single query parameter (?limit=<offset>,<limit>
). For this case, you can use the next_page_token
variable to configure a query parameter with key limit
and value {{ next_page_token['next_page_token'] or '0' }},50
to inject the offset from the pagination strategy and a hardcoded limit of 50 into the same parameter." 5. Another potential solution, as mentioned in the GitHub issue, is to add incremental sync to your stream definition. This seemed to resolve the issue for some users. If these solutions don't work for your specific case, you might need to provide more details about your connector configuration or consider reporting this as a new issue on the Airbyte GitHub repository.