Afternoon again, I am not sure if i have found a ...
# help-connector-development
j
Afternoon again, I am not sure if i have found a bug or not on the cursor pagination on the connector builder. for every new page, it seems to be duplicating the request parameters if you have specified. For instance this is the request for the first page
Copy code
{
  "url": "<https://api.jobadder.com/v2/users>",
  "parameters": {
    "updatedAt": [
      ">2000-01-05T00:00:00Z"
    ],
    "include": [
      "inactive, deleted"
    ]
  },
  "body": null,
  "headers": {
    "User-Agent": "python-requests/2.28.2",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer 86f6f5d182afa48305d9ea85a439d1b9"
  },
  "http_method": "GET"
}
the updatedAt and the include are specified request parameters, on the next page the request looks like
Copy code
{
  "url": "<https://eu2api.jobadder.com/v2/users>",
  "parameters": {
    "updatedAt": [
      ">2000-01-05T00:00:00Z",
      ">2000-01-05T00:00:00Z"
    ],
    "include": [
      "Inactive",
      "Deleted",
      "inactive, deleted"
    ],
    "offset": [
      "100"
    ]
  },
  "body": null,
  "headers": {
    "User-Agent": "python-requests/2.28.2",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer 86f6f5d182afa48305d9ea85a439d1b9"
  },
  "http_method": "GET"
}
and again on the next one its
Copy code
{
  "url": "<https://eu2api.jobadder.com/v2/users>",
  "parameters": {
    "updatedAt": [
      ">2000-01-05T00:00:00Z",
      ">2000-01-05T00:00:00Z",
      ">2000-01-05T00:00:00Z"
    ],
    "include": [
      "Inactive",
      "Deleted",
      "Inactive",
      "Deleted",
      "inactive, deleted"
    ],
    "offset": [
      "200"
    ]
  },
  "body": null,
  "headers": {
    "User-Agent": "python-requests/2.28.2",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer 86f6f5d182afa48305d9ea85a439d1b9"
  },
  "http_method": "GET"
}
also related it doesn't seem to be passing through the page_size either
s
@Josh Jeffries to pass the page size, you need to check the “page size option” and inject it
looking into the bug you are seeing here
j
Hey @Sherif Nada, i have tried it with the page size requester option as well, however, this also has a duplicate on the request, which will then make it fail on the 2nd page (see pic). edit: if i set the page size to 100 and add the page size request option, then this does work and doesn't duplicate the 'limit' request parameter, but if you have any other page side value it does duplicate
a
@Josh Jeffries can you share the YAML representation of your connector? It'll help us troubleshoot the duplicate parameter
j
Hey @Alexandre Girard (Airbyte) Yead not a problem, see attached. i have removed all of my other streams from it and left one is, however, it is pretty much the same for all the streams with different steps and start dates for the incremental sync.
a
thanks! Will look into it. Issue is #23497 for reference
👍 1