https://linen.dev logo
Join Slack
Powered by
HI all! I've written two custom HTTP connectors at...
# help-connector-development
e
HI all! I've written two custom HTTP connectors at this point as I knew they has specialized semantics that wouldn't be a fit for the low-code / no-code world. I'm now trying to add a new API that seemed like a better fit, but I'm running into limits (or lack of knowledge) of the low-code/no-code world and want to make sure I'm not missing something. 1. For pagination, the API returns a "next" token that is a full URL to the next page to fetch (e.g. "next": "https://api.foo.com/thing?page=2"). This "next link" is pretty common in APIs I've seen, but the cursor pagination seems to expect that the field is just a number, not a full blown link. Do the built-in paginators support this URL-based cursor format? 2. For objects, there are additional API calls to get more details about them. So I might do a 
GET /api/foo
to get a list of objects, but then 
GET /api/foo/{id}
will return more information about them (such as additional fields or modification history). In a custom HTTP connector I can easily chain slices to have a stream based off another, is there a way to do this in the low-code/no-code world? Thanks!
👀 1
k
A message from kapa.ai
1. Yes, the built-in paginators support URL-based cursor format. In the case of CursorPagination, you can extract the URL of the next page from the response headers. Here's an example configuration:
Copy code
yaml paginator:    type: "DefaultPaginator"    <...>    pagination_strategy:    type: "CursorPagination"    cursor_value: "{{ headers['urls']['next'] }}"    page_token_option:    type: "RequestPath"
This configuration will send the next request using the URL extracted from the response headers. You can find more information about this in the [Pagination Strategies documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination). 2. In the low-code framework, you can use partition routing to achieve this behavior. You can use the [SubstreamPartitionRouter](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/partition-router#SubstreamPartitionRouter) to create a sub-collection based on the parent-resource ID. This allows you to fetch additional information about the objects by making additional API calls. You can find more information about partition routing in the [Partition Router documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/partition-router). I hope this helps! If you need more guidance, you can refer to the [Low-code connector development tutorial](https://docs.airbyte.com/connector-development/config-based/tutorial/getting-started) for a step-by-step process.
e
Ah, I see that there is a "path" option for where to inject the response, that is what I was missing. I assumed I had to inject it into a query parameter/etc.
Looks like I'll have to go low-code (vs. no-code) to do some of the details fetching, which is not a surprise
OK, now I'm confused. I can see in my response this value:
Copy code
{
  "status": 200,
  "body": {
    "count": 2047,
    "next": "<https://foo.com/api/public/thing/?page=2>",
    "previous": null,
    "results": [
      {
         ...
However, when I attempt to configure this in the no-code builder with next page cursor of 'next' and inject cursor value into 'path', I get this error:
Copy code
Request to <https://foo.com/thing/?page=2> failed with status code 404
Notice that it's missing the 
/api/public
part of the path. I have no idea where it could be getting this abbreviated URL, since it's definitely not in the next token I see in the response. What am I missing?
a
Hi @Eric Schrock thanks for reaching out! This is a known issue with the cursor paginator. You should be able to workaround it by moving the 
/api/public
part of the URL from the base_url to the stream's path
e
Thank you, that works!
🙌 1
Oh and I was also able to get the nested children to work through partitions with a substream type. So looks like this can all be done in no-code after all.
🔥 2
2 Views
Open in Slack
PreviousNext
Powered by