Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

image.png

Hi all,
I have strange situation with pagination, using connector builder.
1. page size = 1000 
2. Start from page = 1
3. Inject page number into outgoing HTTP request → request parametr → page
API return me 1000 records, and not tried get another 1000. It looks odd, because i have page size 1000.

What can be wrong?
I use last build of airbyte (0.5)

Hi Victor, within the low-code builder, when running the Test, we currently limit the number of records returned to 1000 for the request. We apologize  for the confusion but we do this to speed up the time it takes to run a test sync and reduce the number of requests made to target API

I would recommend just testing w/ a smaller page size if you want to verify pagination is working for your connector.

But if you must test over 1000 records, you can change the limit used by the frontend to a higher value <https://github.com/airbytehq/airbyte-platform/blob/main/airbyte-webapp/src/services/connectorBuilder/ConnectorBuilderStateService.tsx#L450|here>. I would caution that this is a change from the default of the connector builder test panel.

oh, thank you, it really helps me.
i can’t test smaller parts because API return me parts by 1000 and i can’t set it smaller.