Hello everyone, I'm creating a custom connector for reading data from Salesforce Reports.
To do that I'm using Salesforce Reports and Analytics
API . This API has no support for pagination and only returns the first 2000 rows from the report.
I'm using simple salesforce library to communicate with the Salesforce Reports api.
I had to write a class in order to create the pagination strategy. I mean, I send the request to the report endpoint sorting the return by the id... and then I keep sending requests to get the next 2000 rows until I get a response with 'all data' property... So I know there's no more data to retrive...
Actually I see no use for the methods next_page_token, path, and other thing in HTTPStream
However it seems to me when we run a python main.py read... the airbyte will use these methods to make the whole thing to work...
I tried to find some samples, documentation in a way I could see clear how to adapt my code... But it's being hard to find that...
Can you share some tips to help im my journey?