https://linen.dev logo
Join Slack
Powered by
I'm building a Front integration. It is paginated ...
# help-connector-development
w
I'm building a Front integration. It is paginated via a 
page_token
query param in a returned URL. Is there a way to only inject the date range query parameters in the first request? I ask because the next path includes the previous query parameters, so after a few paginations the path includes many date range params.
Copy code
"next": "<https://wingspan.api.frontapp.com/events?q[after][0]=1685577600&q[after][1]=1685577600&q[after][2]=1685577600&q[after][3]=1685577600&q[before][0]=1685641246&q[before][1]=1685641246&q[before][2]=1685641246&q[before][3]=1685641246&page_token=2d018a5809eb90d349bc08c52cb1f4988be5b03b8262ad76bb8228d1fe6b9cf6770311505076bbcb11f23f82ce05881a8b015c46b2ebf9419adec233df2863bc>"
j
Could you try the "path" injection option instead of request_parameter? It's named a little confusingly because it will also work with a full URL.
w
@Joe Reuter (Airbyte) To clarify, I was previously using the path injection. The path provided as the next page for pagination already includes the after and before date range query params. Airbyte then adds those query params again at each pagination. After ~100 pages, the path is too long and the sync fails.
j
ah, got it. You can't configure this today unfortunately. Often it's possible to construct the URL for the next page yourself which would work around the problem - is the page token somewhere else in the response as well?
If not I have one other idea - instead of specifying the path in the response object, switch to "Custom" which allows you to define a jinja expression to extract the page token. In the jinja expression use string manipulation to get only the page_token out of the next URL and then set it as the 
page_token
request_parameter
The following expression will probably work (didn't test though):
Copy code
{{ response._pagination.next.split('page_token=')[1].split('&')[0] }}
w
brilliant
Thank you
The page token is only in a query param in a full URL in the response 🤷
Open in Slack
PreviousNext
Powered by