I'm currently working on creating a custom Airbyte...
# replication-ask-ai
i
I'm currently working on creating a custom Airbyte connector to consume data from an API endpoint with a unique pagination structure. The API endpoint URL structure is as follows:
Copy code
<http://example.com/API/v1/orgid/schedule/<year>/r12|example.com/API/v1/orgid/schedule/<year>/r12>
In this structure, I need to dynamically inject the "year" parameter as part of the URL path. The challenge is that the next page token does not appear in the API response. Instead, we must manually increment the previously-used page token if the API response contains data. When no more data is present, we can assume there are no additional pages to retrieve. My initial attempt to define the connector pagination in the manifest.yaml file is as follows:
Copy code
paginator:
  type: DefaultPaginator
  pagination_strategy:
    type: CustomPaginationStrategy
    class_name: "source_example.my_pagination_strategy.CustomYearlyPaginationStrategy"
    year: "{{ config['year'] }}"
However, I would prefer to use a non-custom paginator if possible to simplify the implementation. If that's not an option, I would appreciate guidance on how to implement the custom pagination strategy to achieve this dynamic path change based on the "year" parameter. Any help or examples on how to structure the manifest.yaml file or suggestions on using an existing paginator for this unique use case would be greatly appreciated.
k
New message text here
i
how should I implement the
next_page_token
in the manifest.yaml file to achieve this dynamic path change based on the "year" parameter?
how should I implement the
next_page_token
in the manifest.yaml file to achieve this dynamic path change based on the "year" parameter?