https://linen.dev logo
Join Slack
Powered by
Hi all, I'm trying to build a new stream that inv...
# help-connector-development
a
Hi all, I'm trying to build a new stream that involves pagination. For building requests, I use JSON (free form) like this: 
{
"id": "{{ now_utc() }}",
"method": "SerpstatDomainProcedure.getDomainKeywords",
"params": {
"se": "{{config['region']}}",
"page": "{{next_page_token['next_page_token'] or 1}}",
"size": "{{config['page_size']}}",
"domain": "{{config['domain']}}"
}
}
Our API requires an integer 
page
parameter pointing to the page number with the size of 
size
results per page if I need to walk through all the available results. The 
page
parameter is not returned in the response and is just a serial number of the page. With the JSON above I can't get results from pages 2, 3, and so on. I tried to use built-in Pagination options, but they do not support nested structures like 
params.page
to set the required page number in the request. Is there any way to solve this using JSON (free form) or other connector builder options?
k
A message from kapa.ai
Yes, you can handle this situation using the built-in Pagination options in the connector builder. The Pagination methods supported in the connector builder include Offset Increment and Page Increment, which seem to be suitable for your use case. You can use the "Inject page size / limit / offset into outgoing HTTP request" option in the pagination form. However, if the API has special requirements that can't be handled this way, you can disable injection in the pagination form and use the generic parameter section at the bottom of the stream configuration form to freely configure query parameters, headers, and properties of the JSON body, by using jinja expressions and available variables. Here is an example of how you can set it up:
Copy code
paginator:    type: "DefaultPaginator"    page_size_option:    type: "RequestOption"    inject_into: request_parameter    field_name: size    pagination_strategy:    type: "PageIncrement"    page_size: {{config['page_size']}}    page_token:    type: "RequestOption"    inject_into: "request_parameter"    field_name: "params.page"
In this example, the "page" request parameter value is set to the page to fetch, and the "size" request parameter is set to the number of results per page. The pagination strategy is set to "PageIncrement", which means the page number will increment by 1 for each request until no more pages are available. Please refer to the following documentation for more details: - [Pagination methods](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination) - [Request options](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/request-options) - [YAML Reference](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference) If none of these pagination methods work for your API, you may need to use the [low-code CDK](https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview) or [Python CDK](https://docs.airbyte.com/connector-development/cdk-python/) instead.
j
can you share your whole manifest?
a
@Joe Reuter Is this what you are asking about? 
spec:
type: Spec
documentation_url: <https://example.org>
connection_specification:
type: object
$schema: <http://json-schema.org/draft-07/schema#>
required:
- api_key
properties:
domain:
type: string
order: 1
title: Domain
region:
type: string
order: 4
title: Region
default: g_us
api_key:
type: string
order: 0
title: API Key
airbyte_secret: true
domains:
type: array
order: 5
title: Domains
keyword:
type: string
order: 2
title: Keyword
page_size:
type: integer
order: 3
title: Page size
additionalProperties: true
type: DeclarativeSource
check:
type: CheckStream
stream_names:
- Domain history
streams:
- name: Domain history
type: DeclarativeStream
retriever:
type: SimpleRetriever
paginator:
type: NoPagination
requester:
path: /
type: HttpRequester
url_base: <https://api.serpstat.com/v4/>
http_method: POST
authenticator:
type: ApiKeyAuthenticator
api_token: '{{ config[''api_key''] }}'
inject_into:
type: RequestOption
field_name: token
inject_into: request_parameter
error_handler:
type: CompositeErrorHandler
error_handlers:
- type: DefaultErrorHandler
backoff_strategies:
- type: ExponentialBackoffStrategy
factor: 2
request_headers: {}
request_body_json:
id: '{{ now_utc() }}'
method: SerpstatDomainProcedure.getDomainsHistory
params:
se: '{{config[''region'']}}'
size: '{{config[''page_size'']}}'
sort:
date: asc
domain: '{{config[''domain'']}}'
during_all_time: true
request_parameters: {}
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- result
- data
primary_key:
- date
schema_loader:
type: InlineSchemaLoader
schema:
type: object
$schema: <http://json-schema.org/schema#>
properties:
ads:
type: number
date:
type: string
traff:
type: number
domain:
type: string
visible:
type: number
keywords:
type: number
ad_keywords:
type: number
new_keywords:
type: number
out_keywords:
type: number
down_keywords:
type: number
rised_keywords:
type: number
visible_static:
type:
- 'null'
- number
- name: Domains summary
type: DeclarativeStream
retriever:
type: SimpleRetriever
paginator:
type: NoPagination
requester:
path: /
type: HttpRequester
url_base: <https://api.serpstat.com/v4/>
http_method: POST
authenticator:
type: ApiKeyAuthenticator
api_token: '{{ config[''api_key''] }}'
inject_into:
type: RequestOption
field_name: token
inject_into: request_parameter
error_handler:
type: CompositeErrorHandler
error_handlers:
- type: DefaultErrorHandler
backoff_strategies:
- type: ExponentialBackoffStrategy
factor: 2
request_headers: {}
request_body_json:
id: '{{ now_utc() }}'
method: SerpstatDomainProcedure.getDomainsInfo
params:
se: '{{config[''region'']}}'
domains: '{{config[''domains'']}}'
request_parameters: {}
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- result
- data
primary_key:
- domain
schema_loader:
type: InlineSchemaLoader
schema:
type: object
$schema: <http://json-schema.org/schema#>
properties:
ads:
type: number
traff:
type: number
domain:
type: string
visible:
type: number
keywords:
type: number
prev_date:
type: string
ad_keywords:
type: number
ads_dynamic:
type: number
new_keywords:
type: number
out_keywords:
type: number
down_keywords:
type: number
traff_dynamic:
type: number
rised_keywords:
type: number
visible_dynamic:
type: number
keywords_dynamic:
type: number
- name: Domain keywords
type: DeclarativeStream
retriever:
type: SimpleRetriever
paginator:
type: NoPagination
requester:
path: /
type: HttpRequester
url_base: <https://api.serpstat.com/v4/>
http_method: POST
authenticator:
type: ApiKeyAuthenticator
api_token: '{{ config[''api_key''] }}'
inject_into:
type: RequestOption
field_name: token
inject_into: request_parameter
error_handler:
type: CompositeErrorHandler
error_handlers:
- type: DefaultErrorHandler
backoff_strategies:
- type: ExponentialBackoffStrategy
factor: 2
request_headers: {}
request_body_json:
id: '{{ now_utc() }}'
method: SerpstatDomainProcedure.getDomainKeywords
params:
se: '{{config[''region'']}}'
page: '{{next_page_token[''next_page_token''] or 1}}'
size: '{{config[''page_size'']}}'
domain: '{{config[''domain'']}}'
request_parameters: {}
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- result
- data
primary_key:
- keyword
schema_loader:
type: InlineSchemaLoader
schema:
type: object
$schema: <http://json-schema.org/schema#>
properties:
url:
type: string
cost:
type: number
traff:
type: number
types:
type: array
items:
type: string
domain:
type: string
dynamic:
type:
- 'null'
- number
keyword:
type: string
position:
type: number
geo_names:
type: array
items:
type: string
difficulty:
type: number
concurrency:
type: number
found_results:
type: number
keyword_length:
type: number
region_queries_count:
type: number
region_queries_count_wide:
type: number
version: 0.43.0
metadata:
autoImportSchema:
Domain history: true
Domain keywords: true
Domains summary: true
j
Yes
So the thing that's missing is the pagination configuration
enable it and set it to page increment
You don't really need the 
Inject page number into outgoing HTTP request
section in this case, just set it to something like 
X-page
header so it's out of the way (soon it will be possible to disable it https://github.com/airbytehq/airbyte-platform-internal/pull/7464 )
a
That is exactly what I was missing. I enabled pagination, set it to page increment, followed your advice on the header, changed my code to include 
"page": "{{(next_page_token['next_page_token'] or 0) + 1}}"
instead of 
"page": "{{next_page_token['next_page_token'] or 1}}"
, and it seems like it is walking through the pages now. I will test this with more data but anyway - thank you so much for your help.
Open in Slack
PreviousNext
Powered by