I am attempting to create a custom http connector ...
# help-connector-development
b
I am attempting to create a custom http connector to an endpoint that utilizes pagination. Currently I am able to receive a single record and place it in my destination (Snowflake). However, when I try to utilize the next_page_token function (implementation below) I run into errors. If I run my code from VS Code, everything works as expected, but when I run my docker image in airbyte I consistently get 429s before I receive my first record. Would anyone be able to help me understand why the experience is different and what I should be doing here?
Copy code
def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        return response.json().get("@odata.nextLink")
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, I can provide you with some general guidance on handling pagination with Airbyte connectors. When implementing the
next_page_token
function, you should extract the next page "token" from the latest response. The token can be an ID, a page number, a partial URL, etc. The CDK will continue making requests as long as the
next_page_token
function returns non-
None
results. You can then use this token in the
request_params
and other methods in
HttpStream
to page through API responses [source](https://docs.airbyte.com/connector-development/cdk-python/http-streams). For more advanced pagination scenarios, you can use custom parameter injection and jinja expressions with available variables [source](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination). If you're still having issues, I recommend reviewing the Airbyte documentation on [connector development](https://docs.airbyte.com/connector-development/) and [pagination](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination) for more guidance.