Rishav Sinha
05/31/2023, 6:37 AM<https://mydomain.com/api/v3/endpoint?input_data={>"list_info":{"start_index":1,"row_count":"2000","get_total_count":true}
this is how they expect us to pass data for pagination, can someone guide me on this please ?kapa.ai
05/31/2023, 6:37 AMkapa.ai
05/31/2023, 6:38 AMinput_data
and value {"list_info":{"start_index":{{ next_page_token['next_page_token'] or '1' }},"row_count":"2000","get_total_count":true}}
to inject the start_index from the pagination strategy. Here's an example of how you can configure the pagination: yaml paginator: type: DefaultPaginator page_size: "#/definitions/page_size" limit_option: inject_into: "request_parameter" field_name: "row_count" page_token_option: type: RequestOption inject_into: "request_parameter" field_name: "start_index" pagination_strategy: type: "OffsetIncrement" page_size: 2000
Then, in the generic parameter section, configure the input_data
parameter: yaml request_parameters: - key: "input_data" value: '{"list_info":{"start_index":{{ next_page_token["next_page_token"] or "1" }},"row_count":"2000","get_total_count":true}}'
This configuration should help you implement pagination for your custom connector in Airbyte.Rishav Sinha
05/31/2023, 7:03 AMkapa.ai
05/31/2023, 7:03 AMkapa.ai
05/31/2023, 7:03 AMJoe Reuter (Airbyte)
05/31/2023, 7:36 AMJoe Reuter (Airbyte)
05/31/2023, 7:36 AMJoe Reuter (Airbyte)
05/31/2023, 7:36 AMJoe Reuter (Airbyte)
05/31/2023, 7:37 AMJoe Reuter (Airbyte)
05/31/2023, 7:37 AM{"list_info":{"start_index":"{{ next_page_token. next_page_token or 0 }}","row_count":"2000","get_total_count":true}
Joe Reuter (Airbyte)
05/31/2023, 7:37 AMRishav Sinha
05/31/2023, 7:44 AM