@Andy Yeo (Airbyte) @Alexandre Girard (Airbyte) @Conor Barber (Airbyte) @Erica Struthers (Airbyte) Do we have any example of low code connector where we have implemented Oauth2 authentication, paginator and nested streams. I have tried everything using documentation but its not working. I am stuck at this point where i can fetch all the data but paginator and nest streams dont work. I have raised many tickets and sent questions on slack as well. Let me know in case of any example if we have used the above request in low code connector.

cc @Brian Lai (Airbyte) as I think he knows quite a few api sources (feel free to route onward if this isn’t your domain brian)

hi @Shreepad Khandve do you know what type of pagination strategy your API is using (cursor based, record offsets, page based)? I can try to find a specific example for your type. As for nested streams, unfortunately we don’t currently support that functionality just yet, but we have an issue https://github.com/airbytehq/airbyte/issues/22873 tracking it that you can follow. We’re hoping to prioritize this work soon

@Brian Lai (Airbyte) Thanks for response. So one of my api pagination is based on next link coming in json response but not in headers. Another api pagination is based on page increment, its taking the next page but its not stoping at one point. let me know if we have any example in low code cdk

We should support the next page cursor being in the response. Here is an example of an existing source where we search in the JSON response body, traversing

records

->

cursor

. https://github.com/airbytehq/airbyte/blob/8ff26a38a1a89a2d9e1214c43715477f0a89327d[…]e-integrations/connectors/source-gong/source_gong/manifest.yaml Does this fit w/ your use case?

I tried cursor pagination as well, but its not capturing the next link Below is the response and the pagination i have used - Response =

{..............."links": {"next": "<https://iteratehq.com/api/v1/surveys/5d6e82a775b0********/responses?access_token=*********&page[cursor]=5d6f14e2b27070798fab757d>"}}, "emitted_at": 1680030710894}}

paginator -

paginator:
        type: DefaultPaginator
        pagination_strategy:
          type: "CursorPagination"
          cursor_value: "{{ response.links.next }}"
          stop_condition: "{{ 'links' in response }}"

in this scenario, shouldn’t the

stop_condition

be

"{{ 'links' not in response }}"

? The way it is written we stop pagination when we see that response has a

links

field instead of continuing

Okay understood. Another use case is - response - (page - page number, count- page size, total records )

{
    "status": {
        "code": 200,
        "message": "ok"
    },
    "pagination": {
        "page": 5,
        "count": 10,
        "total": 5191
    },
    "products": [ ...........}

Paginator - (here i was talking about the iteration of next page is working but its bot stoping after fetching all records .)

paginator:
        # type: NoPagination
        type: "DefaultPaginator"
        page_size_option:
          type: "RequestOption"
          inject_into: "request_parameter"
          field_name: "count"
        pagination_strategy:
          type: "PageIncrement"
          page_size: 1
        page_token_option:
          type: "RequestOption"
          inject_into: "request_parameter"
          field_name: "page"

one clarification point, is the

page_size

supposed to be one? that field is supposed to be the number of items per page. it is possible for it to be one, but generally uncommon since most apis return anywhere from 10 to 100ish records per response

When i keep page_size = 1, the page number is iterating from page 1, 2 , 3 n so on. The page size is ideally 150 records per page for my api but when i set 100 or 150 instead of 1 as page size ,the page is not iterating.

I see, I think it continuing could have to do with the fact that your page_size is 1. We decide to stop paginating when the number of records retrieved is <

page_size

. We could just always be receiving the final record and keep on paginating. This is my initial though, do you by chance have any API docs?

Yes, count is basically the product count per page, max is 100 https://core-api.yotpo.com/v1.0/reference/retrieve-all-products

I see, and how are you building this new source connector? is it on a IDE or via our connector builder UI in the airbyte web app?

Using the web app only

no problem at all, would this example work? https://github.com/airbytehq/airbyte/blob/8ff26a38a1a89a2d9e1214c43715477f0a89327d[…]tegrations/connectors/source-square/source_square/manifest.yaml We don’t currently support single use tokens, but we should allow for a refresh flow when the expiry passes

I am checking that on IDE, using read command i am fetching the records for small subset

and is

count=150&page=1

(1,2… 4) working?

I checked the above url in postman as well. When i pass count=150&page=1 , it works but when i pass count = 150 in pagination logic it doesn't work. Page is not iterating like page= 1, 2 ...

Yes that might be easier, but do you mind if we find some time to sync tomorrow instead? Can we meet at 11am PST on 3/29?

Sure, works for me. Lets connect tomorrow then. Thanks alot 👍

Hi Shreepad, can we meet a little later today, I am free at 3pm PST and onward, or if you would like to sync thursday that would work for me as well

Thursday will work for me at same time 11 am pst, thanks 👍

hi shreepad yes, but I thought we were meeting 11am PST. its currently 10:23am PST

Yes, just confirming. Will connect in half an hour then

Hey Shreepad sorry for the delay. Glad to hear the offset/page pagination is working for your low code connectors. I looked into a few of your questions, specifically catching 401s and getting a new token authentication. We do not currently support that, and our main way of performing a refresh is when the token expiry is reached and the refresh flow is run again (as opposed to knowing to refresh when we get a 401). sorry about that. As for the issue with cursor pagination you were seeing, I’m still investigating this one a bit and will let you know what I find

Okay sure, Thanks Brain ! let us know about cursor pagination.

hi shreepad, I’m sorry for the delay here. I had a chance to quickly take a look last week but didn’t come away with a solution. We do have an issue flagged https://github.com/airbytehq/airbyte/issues/23497 that is in our plan to work on soon. It could be related, but I’ll need to double check